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Guidance  for  training  development  le  provided  in  US  Army  Training  and 
Doctrine  Command  (TRADOC)  Pamphlet  350-30,  Interservice  Procedures  for 
Instructional  Systems  Development  (ISO),  August  1975.  Some  of  the  training 
being  developed  and  conducted  does  not  follow  this  guidance,  however, 
because  many  of  the  training  developer*  and  deliverers  have  not  been 
trained  in  the  development  of  a  training  process;  they  are  subject  matter 
experts  who  are  detailed  to  training  development.  Training  is  typically 
developed  under  a  concern  for  what  Is  being  trained.  How  that  training  Is 
developed,  conducted,  and  evaluated  ls~ usually  given  less  attention. 
Because  training  Is  seldom  subjected  to  formal  evaluation.  Ineffective 
training  may  go  uncorrected.  US  Army  Research  Institute  Fort  Knox  Field 
Unit  has  developed  a  system  for  formally  evaluating  the  training  process. 
Its  use  should  measurably  upgrade  training  development,  conduct,  and 
evaluation  in  the  Army. 

f 

Training  Program  Evaluation  (TPR)  Is  a  syatem  for  evaluating  the 
effectiveness  and  efficiency  of  a  training  progrem.  It  incorporates  the 
decision  rules,  data  collection  formats,  and  analysis  procedures  for 
evaluating  the  soundness  of  e  training  plan  as  expressed  In  lesson  plans 
and  training  materials,  evaluating  the  training  and  testing  process, 
identifying  training  and  non-training  cauaee  of  poor  soldier  performance  In 
training,  and  for  recommending  modifications  to  training  and  testing  that 
have  a  high  probabity  of  eliminating  poor  soldier  performance  in  training. 

This  job  aid,  one  of  four  in  the  system,  provides  guidelines  for 
conducting  the  overall  evaluation,  and  Includes  guldence  for  using  the 
other  job  aids.  It  Includes  detailed  procedures  for:  planning  an  evalua¬ 
tion;  training  and  supervising  training  observers;  planning,  conducting, 
and  managing  training  and  testing  data  collection  efforts;  and  translating 
training  and  testing  data  into  recommendations  for  training  program  modifi¬ 
cation. 
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GUIDELINES  FOR  CONDUCTING  A  TRAINING  PROGRAM  EVALUATION  (TYPE) 
BRIEF 


REQUIREMENT 

The  Army  does  not  have  a  .standard  set  of  procedures  for  evaluating  the 
effectiveness  and  efficiency  df  training  programs.  A  need  for  such  standard¬ 
ized  formal  procedures  was  identified  by  the  Army  Training  Study  in  1978. 
Guidance  from  the  Army  Training  Study  specified  the  development  of  procedural 
guides  that  would  not  require  an  analyst  sophisticated  in  educational  tech¬ 
nology,  would  be  applicable  to  established  and  developing  weapon  systems,  and 
could  be  used  in  institutional,  transition,  and  sustainment  environments. 

PROCEDURE 

Training  Program  Evaluation  (TPE)  was  designed  as  a  system  for  evaluating 
the  effectiveness  and  efficiency  of  a  training  program.  The  design  of  the  TPE 
system  was  based  on  civilian  and  military  training  evaluation  literature,  in¬ 
dustrial  practice,  and  the  experience  of  the  research  team.  Included  in  the 
TPE  system  were  procedural  guidelines,  decision  rules,  data  collection  formats, 
and  analysis  procedures  for  planning  and  conducting  a  complete  training  program 
evaluation.  The  guidelines,  decision  rules,  data  collection  formats  and  analy¬ 
sis  procedures  were  designed  to  allow  a  relatively  unsophisticated  evaluator 
to  evaluate  the  soundness  of  a  training  plan  as  expressed  in  the  lesson  plans, 
evaluate  the  training  and  testing  process,  analyze  the  test  score  data,  iden¬ 
tify  training  and  non-training  causes  of  deficient  soldier  performance  during 
training,  and  recommend  modifications  to  training  and  testing  that  have  a  high 
probability  of  eliminating  performance  deficiencies  by  increasing  training 
ef  f  ec  t  ivfenes  s . 

To  assist  the  evaluator  in  knowing  what  to  look  for  during  training  and 
testing,  observable  elements  of  the  training  and  testing  process  were  specified. 
These  elements,  or  items,  included  such  things  as  whether  or  not  everyone  prac¬ 
ticed  the  training  task  to  standard,  if  the  training  aids  specified  by  the 
lesson  plan  were  used,  whether  or  not  tasks  were  demonstrated,  if  testing  was 
contaminated  by  unwarranted  prompting,  etc.  These  items  were  formatted  into  a 
worksheet  and  given  several  field  trials  with  typical  users.  The  major  field 
trial  was  conducted  in  conjunction  with  the  Ml  tank  OT-III. 

Following  the  series  of  field  trials,  lessons  learned  were  compiled  and 
the  items,  guidance,  and  suggested  worksheet  formats  were  finalized.  For  each 
set  of  observable  items  on  the  finalized  worksheets,  program  modifications  cor¬ 
responding  to  those  items  were  devised  to  enable  the  evaluator  to  correct  any 
problems  identified  on  the  worksheets.  Program  modifications  are  therefore 
indexed  to  problems  identified  during  the  training,  testing,  or  in  the  training 
plans  and  materials.  Because  of  the  particular  importance  of  training  objec¬ 
tives,  practice  and  feedback  to  training  effectiveness,  additional  program 
modification  guidance  was  included  for  each  of  these  topics. 

There  are  four  job  aids  in  the  series:  one  for  the  systematic  evaluation 
of  lesson  (or  training)  plans,  one  for  the  structured  observation  of  ongoing 
training  and  testing,  one  for  modifying  ineffective  or  inefficient  training 
programs,  and  this  job  aid  for  use  by  the  analyst  in  planning  and  conducting  a 
training  program  evaluation. 


FINDINGS 

The  TPE  system  has  been  used  in  several  operational  training  program  eval¬ 
uations  and  has  provided  training  process  data  not  heretofore  available.  Users 
have  found  these  data  useful  for  "fixing"  training  problems. 

UTILIZATION 


Preliminary  versions  of  the  TPE  materials  have  been  provided  to  the  Armor 
Center  and  School,  the  Armor  and  Engineer  Board,  and  the  Office  of  Armor  Force 
Management  and  Standardization  (OAFMS)  at  Fort  Knox,  the  Soldier  Support  Center 
at  Fort  Benjamin  Harrison,  the  Artillery  School  at  Fort  Sill,  the  Ordnance 
Center  at  Aberdeen  Proving  Ground,  the  Infantry  School  at  Fort  Benning,  the 
US  Army  Training  and  Doctrine  Conmand  (TRADOC)  Deputy  Chief  of  Staff  for  Train¬ 
ing,  TRADOC  Training  Development  Institute,  and  the  National  Defense  Headquar¬ 
ters  at  Ottawa,  Canada.  Final  versions  have  been  requested.  In  addition,  the 
system  has  been  implemented,  all  or  In  part,  at  Fort  Hood  where  the  TRADOC  Com¬ 
bined  Arms  Test  Activity,  TRADOC  Systems  Analysis  Activity,  the  Armor  Center, 
and  OAFMS  used  it  to  evaluate  the  New  Equipment  Training  for  the  Ml  tank,  at 
Fort  Knox  where  it  has  been  used  to  evaluate  the  Advanced  NCO  course.  Ml  Tank 
Basic  Armor  Training,  M60A3  Basic  Armor  Training,  and  at  Fort  Bliss  where  it 
has  been  used  as  a  baseline  for  a  system  for  evaluating  the  NET  programs  accom¬ 
panying  Air  Defense  Developing  weapon  systems. 
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SECTION  I 


INTRODUCTION 

The  heart  of  any  training  program  is  the  training  objectives  upon  which 
it  is  based.  The  training  objectives  define  what  the  training  program  is  sup¬ 
posed  to  accomplish.  They  describe  the  tasks  and  subtasks  that  the  soldiers 
must  learn  to  perform,  the  conditions  under  which  the  tasks  are  to  be  performed 
and  the  standards  that  the  soldiers  must  meet  in  order  to  be  considered  profi¬ 
cient.  The  purpose  of  the  training  program  is  to  arrange  a  series  of  training 
events  /that  will  enable  the  soldiers  to  accomplish  the  training  objectives. 
Included  in  most  training  programs  are  tests  for  determining  if  the  soldiers 
can  accomplish  the  training  objectives  following  training.  Performance  on  the 
test  following  training  may  be  used  as  an  index  of  the  effectiveness  of  a  train¬ 
ing  program.  Poor  test  performance  suggests  that  there  are  training  program 
deficiencies  that  may  require  changes  in  the  training  program.  For  example  a 
given  training  program  may  be  considered  effective  if,  on  completion  of  each 
task,  80%  of  the  soldiers  perform  each  task  and  subtask  to  the  training  stan¬ 
dard  under  the  training  conditions  the  first  time  they  are  tested.  If  fewer  - 
than  80%  of  the  soldiers  demonstrate  task  proficiency  for  an  objective  the 
first  time  they  are  tested,  the  effectiveness  of  the  training  associated  with 
that  objective  is  called  into  question  and  an  analysis  may  be  required  to 
identify  training  program  deficiencies  and  modify  training  to  correct  those 
deficiencies. 

Training  Program  Evaluation  (TPE)  is  a  systematic  method  for  identifying 
and  correcting  training  program  deficiencies  by  collecting  and  analyzing  in¬ 
formation  on  the  training  objectives,  soldier  test  performance,  and  the  process 
used  in  training  and  testing  the  soldiers.  TPE  includes:  (1)  procedures  for 
planning  the  training  program  evaluation;  (2)  guidance  in  using  and  evaluating 
the  information  provided  in  the  lesson  plans;  (3)  forms  and  procedures  for  ob¬ 
serving  training  and  testing  as  they  are  conducted;  (4)  methods  for  analyzing 
training,  testing,  and  performance  data  in  order  to  identify  training  program 
deficiencies;  and  (5)  guidance  in  modifying  training  programs  on  the  basis  of 
problems  discovered  during  training  program  evaluation. 

TPE  methods  and  procedures  are  described  in  a  series  of  four  user-oriented 
job  aids.  Separate  job  aids  are  used  in  evaluating  lesson  plans  (Research 
Product  (RP)  81-15),  collecting  training  and  testing  process  information  (RP 
81-16),  and  modifying  the  training  program  in  response  to  deficiencies  dis¬ 
covered  during  the  evaluation  (RP  81-17).  This  job  aid  (RP  81-18)  provides 
guidelines  for  conducting  the  overall  evaluation,  including  guidance  in  using 
the  other  job  aids.  The  four  job  aids  are  listed  below,  and  a  brief  descrip¬ 
tion  of  each  is  provided. 

Research  Product  81-15,  A  Job  Aid  for  the  Systematic  Evaluation  of  Lesson 
Plans.  The  basis  for  the  training  process  is  the  lesson  plan,  fhe  lesson  plan 
prescribes  how  training  and  testing  should  be  conducted.  Good  lesson  plans  can 
(but  do  not  always)  lead  to  good  training.  Poor  lesson  plans  almost  never  lead 
to  good  training.  This  Job  Aid  will  help  you  answer  the  question,  "If  the 
training  is  given  in  accordance  with  this  lesson  plan,  is  it  likely  to  be  any 
good?"  It  will  help  you  evaluate  lesson  plans  for  adherence  to  sound  training 
principles. 
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Research  Product  81-16.  A  Job  Aid  for  thia  Structure**  Observation  of  Train- 
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formation  about  practice,  one  must  know  to  look  for  supervised  hands-on 
performance  of  the  tasks  listed  in  the  training  objectives  during  training. 

To  tell  when  practice  has  occurred,  one  must  observe  each  soldier  as  he  or  she 
practices,  recording  how  many  soldiers  received  supervised  hands-on  practice 
for  each  task.  A  practice  event  that  has  been  done  correctly  will  require 
every  soldier  to  practice  each  task  to  standard  on  at  least  one  unassisted 
trial.  Otherwise  the  practice  event  has  not  been  conducted  correctly.  This 
Job  Aid  lists  109  questions  that  can  be  asked  of  an  ongoing  training  program. 

For  each  item,  the  Job  Aid  contains  a  short  paragraph  describing  what  to  look 
for,  how  to  decide  if  it  has  occurred,  and  how  to  decide  if  it  was  all  right. 

Part  of  your  job  as  an  analyst  will  be  to  select  from  those  109  the  items 
you  want  answered  about  the  training  program  you  will  be  evaluating.  You  will 
also  need  to  decide  how  to  best  format  the  items  you  select  on  the  TPE  work¬ 
sheets.  Formats  for  four  worksheets  are  suggested  in  the  Job  Aid.  Worksheet 
1,  Training  Plan,  will  help  you  summarize  the  lesson  plan  so  that  you  can  de¬ 
termine  if  the  training  given  followed  the  lesson  plan  without  having  to  carry 
the  entire  lesson  plan  with  you  when  you  (or  training  observers  who  work  for 
you)  go  out  to  observe  training.  Filling  out  Worksheet  1  will  also  alert  you 
to  possible  problems  when  the  training  is  given.  Worksheet  2,  Training  Environ¬ 
ment,  will  contain  the  items  you  select  to  gather  information  on  the  place 
where  the  training  was  conducted,  the  training  site.  Worksheet  3,  Training 
Observation,  will  contain  the  items  you  select  to  gather  information  on  the 
ongoing  training  process.  Worksheet  4,  Testing  Observation,  will  contain  the 
items  you  select  to  gather  information  on  the  testing  process. 

Research  Product  81-17.  A  Job  Aid  for  Modifying  Ineffective  or  Inefficient 
Training  Programs.  When  you  have  checked  tne  lesson  plans,  observed  the  train- 
ing  and  testing,  and  obtained  the  test  score  sheets  (to  determine  first  time 
", *  rates),  you  may  decide  that,  for  one  or  more  tasks,  there  is  a  problem. 

Your  next  step  is  to  decide  what  probably  caused  the  problem  and  what  to  do 
about  it  so  you  can  tell  the  training  developer  and/or  training  manager  what 
went  wrong  and  make  recommendations  for  changing  the  training  program  to  cor¬ 
rect  the  problem.  For  each  possible  training  program  problem  identified  using 
the  items  in  the  Observer's  Job  Aid  (RP  81-16),  there  is  a  description  of  the 
problem  and  a  corresponding  problem  solution  in  the  Modifications  Job  Aid  (RP 
81-17).  You  can  go  to  the  appropriate  section  of  RP  81-17  and  find  a  descrip¬ 
tion  of  the  probable  problem  and  suggestions,  derived  from  sound  training  and 
training  management  principles,  for  modifying  training  to  eliminate  the  problem. 

And,  finally.  Research  Product  81-18.  Guidelines  for  Conducting  a  Training 
Program  Evaluation  (TPE) .  the  Job  Aid  you  are  now  reading.  This  Job  Aid  will 
help  you  set  up  and  control  the  whole  evaluation  process.  It  has  been  written 
assuming  that  you  are  the  analyst  for  a  TPE  but  have  not  been  formally  trained 
to  evaluate  training  programs.  It  will  provide  you  with  detailed  guidance  in 
planning  and  conducting  the  training  program  evaluation,  including  guidance  in 
using  the  other  job  aids.  The  process  is  complicated  and  cannot  be  conducted 
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automatically.  The  Research  Products  are  called  Job  Aids  but,  in  a  sense,  do 
not  absolutely  fit  the  usual  description  of  a  Job  Aid.  "How  to"  guidance  is 
perhaps  a  better  description  of  these  Research  Products. 

Following  the  procedures  provided  in  these  four  job  aids  will  allow  you 
to  answer  most  of  the  questions  that  training  managers  want  answered  in  evalu¬ 
ating  a  training  program.  However,  there  are  two  major  questions  that  training 
managers  often  ask  about  a  training  program  that  TPE  does  not  address.  TPE 
will  not  answer  the  question  "Are  we  teaching  the  right  things  to  the  right 
standards  under  the  right  conditions?"  That  is  called  "front-end  analysis." 

TPE  does  not  include  front-end  analysis.  It  will  not  answer  the  question,  "Do 
the  skills  and  knowledges  learned  in  this  training  help  the  soldier  do  his  or 
her  job?"  That  is  called  "transfer  of  training."  Front-end  analysis  and 
transfer  or  training  may  need  to  be  considered  in  a  complete  program  evaluation 
they  just  are  not  covered  by  TPE.  What  TPE  is  concerned  with  is  the  question 
of  whether  or  not  good  training  and  testing  procedures  are  consistently  fol¬ 
lowed  throughout  the  training  program. 


SECTION.  II 

EVALUATION AN.  OVERVIEW 

There  is  widespread  agreement  among  the.  Army  leadership  that  training 
programs  should  be  evaluated.  But  there  is  little  agreement  on  what  is  meant 
by  evaluation  and  even  less  agreement  on  ho.w  training  programs  should  be  evalu¬ 
ated.  For  some,  evaluation  is  synomymous  with  testing  soldiers.  Others  think 
of  "validating  instruction"  when  evaluation  is  mentioned.  Still  others  conjure 
up  visions  of  large-scale  tests  and  analyses  such  as  Operational  Tests,  Force 
Development  Tests  and  Evaluations,  and  Cost  and  Training  Effectiveness  Analyses. 
Many  methods  of  evaluation  have  been  and  are  being  used  to  assess  the  effec¬ 
tiveness  of  training  programs.  Some  evaluators  prefer  to  question  soldiers  or 
instructors  about  the  adequacy  of  the  programs.  Others  maintain  that  the  only 
way  to  evaluate  training  is  through  the  u^$  of  carefully  designed  performance 
tests  and  rigorous  statistical  analysis  of  the  data.  Among  those  that  advocate 
performance  testing  as  the  means  to  evaluate  training  programs,  there  is  often 
disagreement  concerning  when  and  under  what  conditions  the  performance  should 
be  measured. 

TPE  as  a  system  for  evaluating  training  programs  does  not  utilize  soldiers' 
or  instructors'  opinions  concerning  the  training,  nor  does  it  rely  on  per¬ 
formance  testing  for  the  bulk  of  its  datf .  TPf  is  a  balanced  approach  to 
training  program  evaluation  combining  planning,  analysis,  observation,  per¬ 
formance  testing  and  program  modification  int?  an  integrated  system  for  evaluating 
training.  TPE  helps  the  training  analyst  to  evaluate  four  major  aspects  of  a 
training  program.  The  four  aspects  are  listed  and  described  below. 

Training  Goals 

First,  evaluation,  as  used  in  TPE,  is  a  process  for  deciding  if  a  training 
program  is,  in  fact,  achieving  its  goals.  The  first  question,  then  is: 

Are  the  goals  of  the  training  program  clearly  stated? 

TPE  cannot  proceed  unless  these  goals  ar.£  clearly  stated.  Goals  are 
usually  stated  as  training  objectives.  A  dearly  stated  training  objective 
specifies  tasks,  conditions,  and  standards  that  are  easily  understood,  measur¬ 
able,  and  achievable  within  the  training  environment.  Major  goals  are  ex¬ 
pressed  as  terminal  objectives  for  the  program  and  interim  goals  are  expressed 
as  intermediate  (or  enabling)  objectives. 

Training  Design 

Once  the  objectives  have  been  clearly  stated,  the  second  question  can  be 
asked . 


Wh#t  is  the  likelihood  that  the  training  program,  as  designed, 
will  achieve  these  goals? 


The  design  of  the  training  program  is  typically  described  in  the  lesson 
plans  for  that  program.  Training  deslqn  includes  the  procedures  used  for 
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training  the  soldiers  to  perform  the  required  tasks  and  for  testing  soldier 
performance  upon  conclusion  of  the  training.  Guidance  in  evaluating  the  train¬ 
ing  design  is  provided  in  Research  Product  81-15 ,  A  Job  Aid  for  the  Systematic 
Evaluation  of  Lesson  Plans.  Other  factors  Influencing  the  effectiveness  of 
the  training  design,  including  personnel,  resource  constraints,  and  training 
and  testing  process  variables  are  described  in  Research  Product  81-17,  A  Job 
Aid  for  Modifying  Ineffective  or  Inefficient  Training  Programs.  If  lesson 
plans  are  not  available,  the  training  design  must  be  evaluated  during  the  de¬ 
livery  of  training.  Evaluating  the  training  design  as  the  training  is  being 
given,  however,  requires  that  each  station  at  which  training  is  conducted  be 
monitored  because  each  instructor  may  deliver  Instruction  of  his  or  her  own 
design. 

While  these  job  aids  provide  some  guidance  in  evaluating  the  training 
design,  you  may  need  to  obtain  the  assistance  of  an  educational  technologist 
if  your  evaluation  indicates  many  design  problems  or  if  lesson  plans  describing 
the  training  design  are  not  available.  You  may  also  need  to  consult  an  educa¬ 
tional  technologist  or  some  other  source  if  you  wish  to  answer  questions  about 
the  training  design  not  covered  by  TPE.  For  example,  these  job  aids  do  not 
help  you  determine  if  the  program  design  is  such  that  it  will  produce  the 
skills  needed  on  the  job.  The  job  aids  can  help  you  determine  if  the  training 
program  as  designed  will  likely  train  those  skills  listed  in  the  training 
objectives. 

Training  Process 

With  the  objectives  clearly  stated  and  with  a  firm  knowledge  of  what  the 
nature  and  sequence  of  training  events  should  be  (the  program  design),  the 
analyst  can  ask: 

Has  the  training  program,  as  designed,  been  faithfully 
implemented? 

This  question  asks  if  the  training  program  being  conducted  is  in  accord¬ 
ance  with  program  design.  Is  there  one  program  being  conducted  or  as  many 
programs  as  there  are  instructors?  TPE  activities  here  find  out,  in  detail, 
what  is  happening  so  that  the  differences  between  "what  is  happening"  and  "what 
should  be  happening"  can  be  clearly  defined  for  training  management  personnel. 

Determining  if  the  training  program  was  conducted  in  accordance  with  the 
program  design  may  provide  sufficient  information  about  the  training  process  in 
some  cases.  However,  if  the  training  design  is  poorly  specified  in  the  lesson 
plans  or  if  the  training  delivered  differs  greatly  from  the  training  design, 
then  additional  Information  is  required  to  answer  the  following  question. 

Are  effective  training  procedures  being  utilized  in  conducting 
the  training  program? 


In  order  to  answer  this  question,  you  must  collect  training  process  in¬ 
formation  as  the  training  is  being  conducted.  Using  forms  and  procedures 
described  in  the  Observer's  Job  Aid  (RP  81-16),  information  is  collected  on  the 
conduct  of  such  training  activities  as  lectures,  demonstrations,  and  practice. 


Training  Products 

Having  evaluated  the  training  goals,  training  design,  and  the  training 
process,  the  final  question  can  then  be  addressed. 

Did  the  training  program  achieve  its  goals? 

This  is  the  usual  evaluation  question  that  is  asked.  Answering  this 
question  involves  measuring  the  product  of  the  program,  the  skills  and  knowl¬ 
edges  of  the  trained  soldiers.  This  is  the  only  question  that  training  manage¬ 
ment  usually  asks  and  training  management  personnel  are  usually  impatient  with 
any  attempt  to  answer  the  first  three  questions  first.  This  question  must  come 
last,  however,  because: 

If  the  goals  are  not  clear,  there  is  no  way  of  knowing  if  the 
goals  have  been  achieved. 

If  the  training  program  is  Implemented  using  a  deficient  training 
design,  the  training  goals  are  not  likely  to  be  achieved. 

If  program  implementation  is  not  in  accordance  with  program  design, 
then  the  effectiveness  of  the  training  may  vary  from  one  day  to  the 
next  and  from  one  training  station  to  the  next.  The  peculiarities 
of  the  current  program  implementation  may  determine  whether  or  not 
the  goals  are  met. 

If  the  training  procedures  being  utilised  in  conducting  the  train¬ 
ing  program  are  not  effective,  the  chances  of  achieving  the  goals 
of  the  program  are  greatly  dinrimished . 

The  most  effective  approach  to  evaluating  a  training  program  is  to  answer 
each  of  the  five  questions  in  turn.  Each  question  answered  provides  useful 
information  for  evaluating  the  training  program  and  makes  the  succeeding  ques¬ 
tions  easier  to  answer.  If  the  training  goals  are  not  clearly  stated,  a  clear 
statement  of  the  goals  must  be  formulated  before  the  training  design,  training 
process,  or  training  products  can  be  evaluated.  If  the  training  design  is  de¬ 
ficient  in  some  way,  it  should  be  modified  prior  to  implementation  of  the  de¬ 
sign  in  the  training  program.  If  the  evaluation  shows  that  program  implementa¬ 
tion  is  not  in  accordance  with  program  design,  the  differences  will  have  to  be 
resolved  before  meaningful  evaluation  can  continue.  Even  when  a  review  of  the 
training  design  Indicates  that  it  is  likely  to  be  effective  and  the  program  is 
implemented  as  designed,  the  procedures  used  during  the  conduct  of  the  program 
should  be  carefully  observed  so  that  any  other  deficiencies  that  might  prevent 
the  achievement  of  the  goals  of  the  training  program  can  be  identified.  If 
observation  of  the  training  process  identifies  further  program  deficiencies, 
changes  in  the  training  program  should  be  made  to  eliminate  the  deficiencies. 
When  each  of  the  first  four  questions  have  been  addressed  and  the  necessary 
program  changes  have  been  made,  the  training  program  is  likely  to  achieve  its 
goals.  Thus,  making  the  proper  response  to  the  first  four  questions  increases 
the  chances  that  the  final  question  will  be  answered  affirmatively  when  the 
product  of  the  training  program  is  measured. 
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Notice  that  TPE  requires  that  program  changes  be  made  whenever  deficiencies 
are  discovered.  In  practice,  however,  training  developers  and  implementers 
tend  to  resist  change.  This  resistance  to  change  can  be  overcome  to  some  ex¬ 
tent  by  forming  a  working  team  consisting  of  yourself  and  representatives  from 
the  training  development  and  training  implementation  agencies.  The  team  members 
should  be  from  the  working  (rather  than  the  management)  level  so  they  can 
readily  determine  what  changes  are  feasible  given  the  constraints  under  which 
they  must  operate.  This  team  can  examine  the  results  of  the  TPE  and  decide 
what  changes  will  produce  the  most  benefit,  which  changes  are  feasible,  and 
which  changes  are  most  likely  to  be  adopted  by  their  agencies. 

Although  TPE  is  more  effective  when  program  changes  are  made  as  each  ques¬ 
tion  is  asked,  practical  considerations  sometimes  prevent  the  necessary  changes 
from  being  made  in  a  timely  manner.  In  such  cases,  you  may  have  to  continue 
the  evaluation,  despite  the  fact  that  you  find  deficiencies  at  each  stage;  the 
goals  are  not  too  clear;  the  design  is  vague;  the  design  has  not  been  faith¬ 
fully  implemented;  and  the  procedures  used  during  training  do  not  conform  to 
accepted  training  practices.  When  this  happens,  you  cannot  stop  when  you  find 
vague  goals,  for  example.  You  will  have  to  complete  the  TPE  and  then  make 
recommendations  across  all  phases. 

The  following  outline  describes  the  sequence  of  tasks  that  need  to  be  com¬ 
pleted  to  conduct  a  full  TPE.  More  detail  in  these  steps  follows  in  this  job 
aid. 

Plan  the  Training  Program  Evaluation. 

1.  Background  on  TPE.* 

See:  Introduction.  Section  I,  RP  81-18. 

Evaluation:  An  Overview.  Section  II,  RP  81-18. 

2.  Get  Information  on  the  Training  Program. 

See:  Planning  the  Evaluation.  Section  III,  RP  81-18. 

Describe  and  Evaluate  the  Training  Program  as  Planned. 

1.  Complete  Worksheet  1,  Training  Plan,  for  Each  Block  or  Module. 

See:  Filling  out  Worksheet  1,  Training  Plan.  Section  IV,  RP  81-18. 

2.  Evaluate  the  Training  Design  as  Described  in  the  Lesson  Plans. 

See:  A  Job  Aid  for  the  Systematic  Evaluation  of  Lesson  Plans,  RP  81-15. 


*The  analyst  should  be  familiar  with  all  four  job  aids  before  attempting  to 
conduct  a  TPE. 


Observe  the  Training  Program  as  Conducted. 


1.  Train  Observers  (if  necessary). 

See:  Conducting  a  Workshop  for  Training  Observers.  Section  IV,  RP  81-18. 

2.  Describe  Actual  Training  Environment. 

See:  Training  Observation  Worksheets.  Section  II,  RP  81-16. 

3.  Observe  Ongoing  Training. 

See:  Training  Observation  Worksheets.  Section  II,  RP  81-16. 

Collecting  Data  During  Practice  Events.  Section  V,  RP  81-18. 

4.  Observe  Testing. 

See:  Training  Observation  Worksheets.  Section  II,  RP  81-16. 

Check  for  Soldier  Performance  Deficiencies. 

1.  Analyze  the  Testing  Results. 

See:  Analyzing  Test  Score  Data.  Section  VI,  RP  81-18. 

Matching  Tasks,  Conditions,  and  Standards  Across  Lesson  Plans, 
Training,  and  Testing.  Section  VII,  RP  81-18. 

Outline  Probable  Causes  of  Performance  Deficiencies. 

1.  Examine  All  Evidence. 

See:  Conducting  the  Training  Program  Evaluation.  Section  VIII,  RP  81-18. 
Report  Results  of  TPE  and  Make  Recommendations. 

1.  Report  Results  of  TPE. 

See:  Conducting  the  Training  Program  Evaluation.  Section  VIII,  RP  81-18. 

2.  Make  Recommendations. 


See:  A  Job  Aid  for  Modifying  Ineffective  or  Inefficient  Training  Progr< 
RP  81-17. 
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SECTION  III 


PLANNING  THE  EVALUATION 

The  conduct  of  any  training  program  evaluation  rests  solidly  on  the 
analyst  having  free  access  to  the  following  seven  items.  If  you,  as  the 
analyst,  do  not  have  free  access  to  any  one  of  these,  you  will  not  be  able  to 
conduct  a  very  useful  TPE.  The  requester/user  of  the  TPE  must  be  made  aware 
of  this  at  the  beginning  of  the  planning  phase. 

1.  A  list  of  the  training  objectives  and  sub-objectives  to  be  met  by  the 
course  that  Includes  each  task  and  subtask  and  the  associated  training  condi¬ 
tions  and  standards. 

2.  All  the  lesson  plans  for  the  course  Including  a  copy  of  each  practical 
exercise  and  test. 

3.  Copies  of  all  the  soldier  hand-outs  for  the  course  (TMs,  FMs,  work¬ 
sheets,  etc.). 

4.  Access  to  all  training  and  testing  events.  "Access"  means  getting 
close  enough  to  what  is  going  on  to  see  and  hear  it  all.  Only  established 
safety  considerations  should  be  allowed  to  interfere  with  this  access. 

.  5.  Access  to  the  soldiers  and  instructors  so  that  information  can  be  col¬ 
lected  on  their  reactions  to  the  training.  Free  access  must  be  possible,  that 
is,  observers  must  be  able  to  talk  to  either  soldiers  or  instructors  without 
representatives  of  the  respective  chain-of -command  being  present. 

6.  Copies  of  all  completed  score  sheets  produced  during  any  practical 
exercise  or  test.  This  includes  the  results  of  any  exercise  during  collective 
training  (squad,  crew,  section,  platoon,  etc.). 

7.  Complete  information  on  all  training  program  changes  that  are  made 
during  the  evaluation  including  revisions  of  the  lesson  plans,  training  mate¬ 
rials,  or  tests. 

A.  Meet  with  the  Requester /User  of  the  TPE. 

Set  up  a  meeting  with  the  agency  for  which  you  are  doing  the  TPE.  If  you 
are  doing  the  TPE  for  your  supervisor,  then  this  meeting  is  with  him  or  her. 

Use  this  meeting  to  outline  the  seven  needs  listed  above.  Explain  the  TPE 
process  to  the  user  so  that  time  and  resource  requirements  are  understood  by 
all  parties  concerned  (it  is  assumed  that  you  have  read  all  four  job  aids). 

You  will  probably  be  tasked,  at  the  outset,  to  do  a  TPE  with  insufficient  time 
and  personnel.  You  must  explain  that  observers  are  needed  to  cover  the  train¬ 
ing  and  testing  events.  TPE  is  a  data  based  system  for  which  data  on  the  train¬ 
ing  process  is  required.  Explain  that  these  observers  will  be  needed  some  time 
in  .i«lv<*tu*tf  i>»'  the  mail  of  training  so  t lint  they  can  be  trained.  Explain  that 
not  everyone  makes  a  good  observer  so  you  may  have  to  start  out  with  more  per¬ 
sonnel  than  you  need.  This  is  all  negotiable,  of  course,  but  the  usefulness 
of  the  results  will  be  directly  related  to  the  support  you  receive. 
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Have  a  representative  of  the  training  development  agency  and  the  depart¬ 
ment  actually  giving  the  training  to  the  soldiers  at  this  meeting  so  your 
activities  during  the  training  program  are  understood  by  the  instructors. 
Strongly  emphasize  the  point  that  TPE  is  not  a  system  for  evaluating  the  in¬ 
structors.  The  instructor  is  an  instructional  medium,  a  way  of  getting  in¬ 
formation  to  the  soldiers.  "Instructors"  are  not  the  "training  program."  The 
program  is  much  more.  A  glance  at  the  categories  in  RP  81-17  will  confirm  that 
few  of  the  potential  problems  with  training  can  be  ascribed  to  the  instructors. 
You  will  have  to  spend  some  time  on  this  because  instructors  have  been  taught 
by  past  practices  that  training  evaluators  spell  trouble  for  them.  You  and 
your  observers  will  start  out  as  "trouble"  and  will  have  to  work  to  change 
this  image.  When  the  training  to  be  evaluated  was  developed  by  the  Instructors 
who  deliver  it,  you  as  the  evaluator  become  even  more  threatening.  In  such 
cases,  you  should  take  special  pains  to  assure  the  instructors  that  you  will 
be  working  with  them  directly,  rather  than  with  their  supervisors,  to  obtain 
the  best  training  program  possible. 

Keep  a  record  of  this  meeting.  The  list  of  items  below  should  be  filled 
in  during  the  meeting  or  as  a  result  of  the  meeting. 

1.  Date.  The  meeting  date. 

2.  Participants.  The  names  of  the  participants,  their  agencies,  what 
connection  the  agency  has  to  the  TPE,  and  their  phone  numbers. 

3.  Course.  The  name  of  the  course  you  will  be  evaluating. 

4.  Purpose  of  Course.  Get  a  copy  of  the  document  that  states  the  purpose 
or  goal  of  the  course.  You  can  use  this  to  check  on  whether  the  events  in  the 
course  are  all  related  to  this  purpose  or  goal.  This  purpose  should  be  clear 
and  unambiguous.  If  the  purpose  is  not  clear,  you  will  not  be  able  to  tell  if 
the  training  is  directed  toward  the  goal. 

5.  Primary  Audience.  For  what  kind  of  soldier  was  the  course  designed? 
This  should  be  clearly  stated  by  the  training  developer.  If  it  is  not  clear 
exactly  who  the  training  has  been  designed  for,  how  can  the  training  events  be 
designed  to  fit  a  particular  audience? 

6.  Expected  Entry  Skills/Rnowledges.  Training  developers  write  for  a 
particular  audience.  In  all  cases  they  have  a  particular  kind  of  soldier  in 
mind.  Soldiers  coming  to  the  course  are  assumed  to  have  some  prior  skills  and 
knowledges  that  are  necessary  for  performance  but  do  not  have  to  be  taught  dur¬ 
ing  this  course.  Find  out  what  these  expected  skills/knowledges  are.  You  may 
have  to  dig  a  little.  This  piece  of  Information  has  probably  not  been  generated 
before.  It  will  probably  be  a  "new"  question.  You  will  use  this  information 

to  check  on  whether  these  skills/knowledges  are,  in  fact,  held  by  the  soldiers. 
If  you  find  instructors  covering  these  skills /knowledges  in  any  but  very  general 
review  sessions,  you  will  have  to  find  out  why  (since  they  are  already  supposed 
to  know  these  things  before  coming  to  this  course) .  If  the  soldiers  are  not 
knowledgeable,  the  training  needs  to  be  redesigned.  If  they  are,  the  instruc¬ 
tors  are  wasting  valuable  training  time. 
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7.  User.  Name  of  the  agency  that  has  "chartered"  the  TPE  and  the  name 
and  phone  number  of  the  user  Point  of  Contact  (POC). 

8.  Training  Schedule.  Get  a  copy  of  the  preliminary  training  schedule 
and  arrange  to  get  copies  of  all  additional  schedules  and  schedule  changes  as 
they  are  produced. 

9.  Contacts.  Get  the  names  and  phone  numbers  of  the  POCs  from  the  train¬ 
ing  development  agency  and  the  Instructor  group. 

10.  Constraints.  List,  in  some  detail,  all  the  constraints  that  will 
limit  your  activities  during  the  TPE.  Include  such  things  as  too  little  time, 
too  few  observers,  not  enough  time  between  expected  receipt  of  lesson  plans 
and  observation  of  training,  limited  access  to  some  phases  of  collective  train¬ 
ing,  difficulty  in  getting  "raw"  test  score  data,  limited  clerical  support, 
etc.  Forewarned  is  forearmed!  Your  goal  here  is  to  not  be  surprised  later 
when  constraining  factors  show  up.  If  you  know  about  them  early,  maybe  you 
can  work  around  them. 

11.  Resource  Materials.  Make  a  list,  in  detail,  of  all  the  documents 
you  need  (lesson  plans  by  title,  test  score  sheets,  soldier  hand-outs,  etc.), 
who  you  will  get  each  one  from,  when  you  can  get  each  one  and  when  you  actually 
got  each  one,  and  a  space  for  when  you  get  each  change.  Expect  a  lot  of 
changes. 

12.  Completion  Date.  When  will  your  final  report  be  due.  Note  that  this 
TPE  system  suggests  that  you  analyse  each  "block"  or  "module"  separately  as 
soon  as  the  block  or  module  and  the  associated  test  is  over.  The  final  report 
is  a  formality  but  it  still  takes  time. 

13.  Contingencies.  List  any  events  or  constraining  factors  that  will 
affect  your  meeting  the  completion  date  (particularly  lack  of  resources). 

14.  Previous  Evaluations.  Find  out  if  there  have  been  any  previous  evalu¬ 
ations  of  all  or  any  part  of  this  course.  If  there  have,  try  to  get  copies  of 
all  the  information  that  may  be  available.  You  can  use  this  Information  to 

see  if  any  problems  were  identified  and  solution  strategies  offered.  You  can 
then  check  to  see  if  these  problems  still  exist,  if  the  solution  strategies 
were  followed,  and  if  the  solution  strategies  that  were  followed  have  eliminated 
the  problem. 

15.  Comparative  Study.  Find  out  if  the  performance  of  the  soldiers  in 
this  course  is  to  be  compared  to  the  performance  of  the  soldiers  in  some  other 
course  or  earlier  version  of  this  course.  If  you  are  expected  to  compare  two 
or  more  training  methods,  get  help  from  someone  trained  to  conduct  educational 
research.  The  problems  in  comparing  two  training  methods  or  conducting  research 
to  see  if  training  produces  better  job  performance  in  units  are  beyond  the 
scope  of  this  TPE  system. 

16.  Soldier  Selection  Standards.  On  what  basis  will  soldiers  be  selected 
to  attend  this  course?  You  can  use  this  information  to  find  out  if  the  soldiers 
in  the  course  actually  meet  or  exceed  the  selection  standards.  If  the  soldiers 
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do  not  meet  the  selection  standards,  then  they  are  the  "wrong"  audience.  Poor 
performance  may  be  more  a  factor  of  this  failure  to  meet  the  selection  standards 
than  problems  with  the  course. 

17.  Entry  Test.  Find  out  if  the  expected  entry  skills/knowledges  are 
going  to  be  assumed  or  tested  for.  If  there  is  an  entry  test,  get  a  copy  and 
arrange  to  have  observers  at  the  test. 

18.  Instructor  Selection  Standards.  On  what  basis  will  Instructors  be 
selected?  Find  out  if  instructor  training  is  required.  Such  training  is  re¬ 
quired  more  often  than  it  is  given.  Many  training  problems  can  be  traced  hack 
to  a  failure  on  the  part  of  training  management  to  properly  train  the  instruc¬ 
tor  group. 

You  may  not  be  able  to  obtain  all  of  the  Information  listed  above 
during  the  course  of  a  single  meeting.  However,  you  should  obtain  this  informa¬ 
tion  in  the  planning  phase  as  early  as  possible  and  certainly  before  the  train¬ 
ing  program  begins. 

B.  Select  Tasks  to  be  Evaluated. 

The  user  may  either  ask  that  you  evaluate  all  tasks  or  ask  that  a  sample 
of  the  tasks  be  evaluated.  If  the  user  wants  all  tasks  evaluated,  you  must 
make  sure  that  the  user  is  willing  to  commit  the  necessary  personnel  resources 
to  cover  all  the  tasks.  If  only  some  of  the  tasks  are  to  be  evaluated,  the 
user  should  select  which  tasks  are  to  be  evaluated.  If  a  sample  of  tasks  are 
to  be  used,  you  must  make  it  clear  to  the  user  from  the  beginning  that  you  will 
only  provide  evaluation  Information  on  those  tasks  selected  and  you  cannot 
vouch  for  the  effectiveness  of  training  for  any  task  that  has  not  been  evaluated. 

C.  Prepare  a  Schedule  of  Events  for  the  Evaluation. 

Outline  your  tasks  for  completing  the  evaluation.  This  task  has  not  been 
called  "outline  a  milestone  schedule"  because  such  schedules  are  usually  not 
developed  in  enough  detail.  Be  as  detailed  as  you  can  be  and  update  regularly. 

List  the  tasks  that  you  need  to  do  in  order  to  perform  the  TPE. 

List  each  meeting  that  you  know  will  be  required. 

List  each  item  that  you  know  will  be  needed. 

Catalogue  the  resources  you  know  you  will  need. 

Schedule  workshops  for  training  the  observers. 

List  the  training  modules  or  tasks  that  need  to  be  observed  during 

training. 

List  the  tests  that  need  to  be  observed. 


List  the  briefings  that  you  know  will  need  to  be  given. 

List  the  reports  that  you  know. will  be  required. 

Estimate  when  each  task  will  occur,  how  much  time  it  will  take,  what  re¬ 
sources  you  have  to  satisfy  the  requirement,  and  who  will  do  what  on  each  task 
Be  prepared  for  frequent  changes. 


SECTION  IV 


PREPARING  THE  TPE  WORKSHEETS 
Preparing  Worksheet  1.  Training  Plan 

Worksheet  1,  Training  Plan,  serves  a  dual  purpose.  The  task  of  filling 
it  out,  of  digging  the  needed  information  out  of  lesson  plans,  training  out¬ 
lines,  etc.,  will  make  you  familiar  with  the  sequence  of  events  that  are  sup¬ 
posed  to  happen  according  to  the  plan.  You  will  find  out,  for  example,  if  the 
training  objective  specifies  what  the  soldiers  will  "do"  after  training,  what 
terns  and  concepts  are  being  introduced,  whether  a  demonstration  is  planned, 
what  sequence  of  activities  is  planned  for  the  practice  events,  etc.  You  will 
be  able  to  make  a  determination  of  how  good  the  training  design  is  from  a  com¬ 
parison  of  what  is  in  the  lesson  plan  with  the  questions  in  RP  81-15  and  t*\e 
guidance  in  RP  81-17.  In  addition  to  these  side  benefits  to  the  analyst,  the 
completed  Worksheet  1  provides  a  convenient  outline  for  the  observers  to  follow 
and  on  which  they  can  record  whether  or  not  training  proceeded  in  accordance 
with  the  plan.  Deviations  from  the  plan  can  be  identified  in  detail  rather 
than  globally  from  a  general  impression.  A  blank  Worksheet  1  is  in  Appendix  A. 
An  example  of  a  completed  Worksheet  1  is  in  Appendix  A,  RP  81-16. 

Generalized  instructions  for  completing  a  Worksheet  1  follow  below. 

Class/Lesson  Title 

Training  Observer _ _ _ _ _ _ 


The  Class/Lesson  Title  is  the  name  of  the  block  of  instruction  that  is  to 
be  observed.  Examples  are:  Assembly/Disassembly  of  the  M240  Machinegun,  Set¬ 
ting  Headspace  and  Timing  on  the  M2  HB  Machinegun,  Conduct  of  Fire  (M60A1  Tank), 
First  Aid,  Supply  Management,  Operation  of  the  PRC-77  Radio  and  Entering/ 
Leaving  the  Net.  Some  of  these  are  long  blockB  of  instruction  and  some  are 
short.  The  "block"  of  instruction  on  which  an  evaluation  will  be  made  is  de¬ 
termined  by  how  the  training  agency  blocks,  or  forms  modules,  of  instruction. 
Accept  their  blocks  as  given.  For  each  block,  then,  you  will  have  one  Work¬ 
sheet  1  (but  as  many  Worksheets  2-4  as  you  have  observers) .  Each  observer  will 
get  his  or  her  own  copy  of  this  completed  Worksheet  1.  The  entry  in  "Class/ 
Lesson  Title"  is  taken  from  the  lesson  plan.  "Training  Observer"  is  filled  in 
by  the  observer  when  he  or  she  takes  his  or  her  copy  to  the  training  site. 

1.  List  under  the  appropriate  heading  below  the  training  aids,  equipment, 
and  materials  that  will  be  used  during  this  lesson. 

Training  Equipment  Training  Materials  Training  Aids 


The  lists  required  by  Item  1  are  obtained  directly  from  the  lesson  plan. 
"Training  Equipment"  is  defined  to  be  any  equipment  used  by  the  soldiers  during 
training.  Examples  of  Training  Equipment  include  weapon  systems,  tools,  test 
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equipment,  simulators,  etc.  "Training  Materials"  are  all  printed  materials 
(other  than  job  aids)  provided  to  the  soldiers  for  their  use  during  training. 
Training  materials  include  such  things  as  technical  manuals,  workbooks,  and 
handouts.  "Training  Aids"  are  any  equipment  or  materials  used  by  the  instruc¬ 
tor  for  the  purpose  of  making  the  training  process  easier  or  more  effective. 
Examples  of  Training  Aids  include  slides,  films,  projectors,  audio  and  video 
equipment,  charts,  magnetic  boards,  chalkboards,  models,  mockups,  sand tables, 
etc.  Note  that  the  term  "Training  Aid"  is  reserved  for  equipment  used  by  the 
instructor.  All  equipment  used  by  the  soldiers  (which  may  also  be  used  by  the 
Instructor)  is  referred  to  as  training  equipment. 


[« 


List  any  job  aids  that  will  be  provided  to  soldiers  and  used  during 
this  training. 


Job  aids  are  step-by-step  directions  to  be  used  in  the  actual  performance 
of  the  tasks  on-the-job.  Job  aids  tell  the  soldier  when-to-perform  and  how  to 
perform. 


[» 


From  the  lesson  plan,  briefly  describe  the  characteristics  required 
the  training  site.  Include  how  the  site  should  be  prepared  for  training 


of 
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List  here  any  requirements  that  the  training  site  should  meet  to  Include 
preparation  of  the  site  for  training  (see  Appendix  A,  RP  81-16  for  an  example). 
Site  requirements  Include  such  things  as  bleachers,  an  Indoor  classroom,  roll¬ 
ing  terrain,  a  laser-safe  range,  etc.  Site  preparation  can  include  such  things 
as  certain  tools  being  available,  certain  malfunctions  being  Introduced  into 
equipment,  specifications  for  preparing  a  concurrent  training  area,  etc. 


I> 


List  safety  precautions  that  should  be  emphasized  and  followed  during 
this  training. 


_ 


List  all  safety  precautions  here.  This  will  help  the  observer  check  to 
see  if  the  required  safety  precautions  are  followed  and  will  also  help  document 
any  constraints  on  observation  Imposed  by  safety  requirements. 


TRAINING  OBJECTIVES 


Task  No 


Objective 


Source 


Part  II  of  Worksheet  1  is  the  observer's  main  source  for  information  about 
the  training  objectives.  Whenever  an  observation  requires  that  the  observer 
know  what  the  tasks/subtasks,  conditions,  or  standards  are  that  are  associated 
with  a  particular  training  objective,  Part  II  of  the  Training  Plan  Worksheet 
is  a  ready  source  of  this  information. 


"Task  No"  can  be  used  to  record  an  official  task  number  or  you  can  use  this 
coluan  to  sequentially  number  the  tasks  and  subtasks  in  the  block  of  instruction 
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5.  TEST  (Test).  Tasks  are  performed  for  evaluation  purposes,  per¬ 
formance  is  scored,  and  scores  are  recorded. 

You  and  your  training  observers  will  communicate  better  if  you  do  not  try 
to  have  too  many  different  kinds  of  training  events.  The  five  above  are  of¬ 
fered  as  being  reasonably  all  inclusive.  If  a  lesson  plan  describes  a  training 
event  that  does  not  fit  any  of  these  descriptions,  pick  the  closest  event  title 
and  add  some  comments  in  the  "Events"  column  to  help  the  observers  Identify  it 
when  it  happens. 

The  "Events"  column  is  where  you  describe  the  event  in  enough  detail  so 
that  the  observer  can  identify  it  when  it  happens  (or  knows  it  did  not  happen) . 
See  Appendix  A  of  RP  81-16  for  an  example. 

For  TERM  events,  list  the  terms  and  concepts  being  introduced  so  the  ob¬ 
server  can  check  them  off  as  they  occur. 

For  DEMO  events,  list  the  task  (subtask,  step)  being  demonstrated  and  any 
special  instructions  to  the  instructor  regarding  the  demonstration. 

For  PRAC  events,  list  the  task  (subtask,  step)  being  practiced  and  any 
special  instructions  to  the  instructor  or  assistant  instructors  regarding  the 
practice  event.  If  the  lesson  plan  contains  an  annex  listing  the  activities 
that  the  assistant  instructor  is  to  have  each  soldier  complete,  attach  the 
annex  rather  than  duplicating  it  on  this  Worksheet. 

Remember  that  the  purpose  of  this  column  is  to  describe  the  training  event 
for  the  observer,  with  as  few  words  as  possible,  in  language  that  the  observer 
will  understand. 

The  column  labeled  "Occur  Yes/No"  is  for  the  use  of  the  observer  in  re¬ 
cording  whether  or  not  the  events  happened.  You  might  also  suggest  to  your 
observers  that  they  jot  down  the  time  for  each  "Yes"  (or  related  group  of  yes 
checks).  This  will  provide  a  record  of  the  amount  of  time  allocated  to  each 
objective  and  event. 

The  "Commerts"  column  is  primarily  for  the  use  of  the  observer  in  record¬ 
ing  what  happened  when  deviations  from  the  planned  events  (and  the  sequencing) 
occur.  The  observer  should  put  enough  information  in  here  to  let  you  know  what 
happened  during  these  deviations. 

Worksheet  1  is  suggested  as  a  substitute  for  the  actual  lesson  plan  for 
the  use  of  the  observer.  By  taking  the  time  to  complete  Worksheet  1  you  make 
it  much  easier  for  the  observer  to  determine  if  the  lesson  plan  was  followed 
during  training,  thereby  increasing  the  objectivity  of  the  data  that  you 
receive. 

Constructing  Worksheets  2,  3,  and  4 

Using  the  items  described  in  RP  81-16,  you  can  construct  Worksheets  2,  3, 
and  4.  Table  1  in  RP  81-16  lists  possible  items  to  be  included  in  Worksheet  2, 
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the  Training  Environment  Worksheet.  Tables  2,  3,  and  4  in  RP  81-16  list  items 
that  can  be  used  in  constructing  Parts  I,  II,  and  III,  respectively  of  Work¬ 
sheet  3,  the  Training  Observation  Worksheet.  Similarly  items  for  building 
Worksheet  4  are  described  in  Table  5  of  RP  81-16. 

In  constructing  these  worksheets  you  will  need  to  decide  two  things: 

(1)  How  should  the  worksheets  be  formatted  to  make  the  worksheets  easy  to  use 
and  provide  sufficient  information  for  the  analysis?  and  (2)  which  of  the  items 
listed  in  Tables  1  through  5  of  RP  81-16  should  be  included  on  each  worksheet? 
The  following  paragraphs  provide  guidance  in  making  these  decisions. 

Appendices  B,  C,  and  D  in  RP  81-16  give  sample  formats  for  Worksheets  2, 

3,  and  4,  respectively.  Notice  that  the  sample  format  is  the  same  from  one 
worksheet  to  the  next;  only  the  worksheet  items  and  headings  differ.  The  head¬ 
ings  on  the  sample  worksheets  are  designed  so  that  only  essential  information 
is  asked  for  on  each  worksheet.  Thus  a  particular  piece  of  information  gener¬ 
ally  does  not  appear  in  more  than  one  heading;  the  exception  to  this  is  the 
class  title,  which  is  used  as  a  bookkeeping  device  to  match  data  pertaining  to 
the  same  block  of  instruction.  The  sample  format  shown  in  RP  81-16  is  recom¬ 
mended  because  of  its  simplicity  and  capacity  to  capture  objective  data.  If 
you  find  this  format  inconvenient,  you  may  modify  the  format  to  better  fit 
your  application.  It  is  strongly  recommended,  however,  that  you  retain  the 
requirement  to  answer  each  question  with  a  "YES"  or  "NO"  response,  and  refrain 
from  making  any  format  change  that  requires  observers  to  submit  subjective 
judgments  about  or  rate  the  training  that  they  observe.  Experience  with  having 
observers  make  judgments  about  the  training  or  rate  the  training  events  has 
shown  that  the  data  provided  by  such  methods  tends  to  be  highly  subjective  and 
of  very  little  value  in  evaluating  a  training  program.  Note  that  the  "Comment*** 
column  Included  on  the  sample  worksheets  is  not  provided  for  the  observer's 
subjective  judgments  or  opinions  about  training;  the  column  is  provided  to  al¬ 
low  the  observers  to  provide  additional  detail  about  what  they  observed. 

After  you  decide  on  a  format,  you  must  select  the  Items  to  be  included  on 
each  worksheet  from  the  items  listed  in  the  appropriate  table  of  RP  81-16. 

Item  selection  is  one  of  the  most  important  tasks  that  you  will  perform,  so 
you  should  take  special  care  in  making  the  selection.  There  are  several  things 
that  you  should  consider  in  selecting  the  items.  Among  these  are  the  item  dif¬ 
ficulty,  the  capabilities  of  the  training  observers,  the  depth  of  the  evaluation 
required,  and  coverage  of  the  major  training  events.  The  items  in  RP  81-16  are 
divided  into  two  levels  of  difficulty  -  items  requiring  easily  made  observa¬ 
tions  that  do  not  involve  judgment  calls  on  the  part  of  the  observer  and  items 
requiring  observations  that  are  difficult  to  make  or  involve  judgment  calls  on 
the  part  of  the  observer.  The  latter  more  difficult  items  are  designated  with 
an  asterisk  in  RP  81-16.  How  many  difficult  items  you  should  include  depends 
on  the  skill  and  capabilities  of  the  training  observers,  the  depth  of  analysis 
required  for  the  training  program  being  evaluated,  and  how  broadly  you  wish  to 
cover  the  training  events.  If  your  observers  have  had  little  experience  using 
TPE  or  you  feel  that  their  skills  as  observers  are  limited,  you  may  want  to 
minimize  the  number  of  difficult  items  included.  On  the  other  hand,  if  you 
want  to  analyze  the  training  program  in  depth,  you  will  need  to  include  many 
of  the  more  difficult  items.  In  order  to  provide  adequate  coverage  of  the 


training  events,  some  o£  the  difficult  items  in  addition  to  the  easier  items 
will  need  to  be  included  on  the  worksheets.  The  items  in  RP  81-16  are  ar¬ 
ranged  by  groups  of  related  items.  To  provide  good  coverage  of  the  training 
program,  you  should  generally  Include  items  from  each  group.  For  example,  all 
items  relating  to  demonstration  of  tasks  are  grouped  together,  as  are  all  items 
relating  to  feedback.  Generally,  it  is  advisable  to  include  some  items  on 
demonstration,  some  on  feedback,  and  some  from  the  other  groups  listed  in 
RP  81-16.  It  is  especially  important  that  the  items  chosen  provide  adequate 
coverage  of  such  major  training  events  as  the  training  objectives,  terminology, 
demonstration,  practice  and  testing.  To  the  extent  that  these  training  events 
are  not  adequately  covered  by  the  worksheet  items,  the  usefulness  of  TPE  as  an 
evaluation  tool  will  be  diminished. 

Before  using  the  worksheets  that  you  constructed  to  evaluate  an  entire 
training  program,  you  should  first  pilot  the  worksheets  against  a  block  of  in¬ 
struction  to  ensure  that  the  worksheets  yield  the  kinds  of  information  that 
you  need.  Then,  pilot  the  worksheets  in  the  TPE  workshop  to  determine  if  the 
worksheets  can  be  used  by  the  training  observers  who  will  collect  the  data 
during  the  training  program.  You  may  find  it  necessary  to  add  or  delete  items 
on  the  basis  of  these  pilot  tests. 


SECTION  V 


If  Che  training  program  you  are  evaluating  has  training  going  on  at  sev¬ 
eral  different  sites  at  the  same  time,  and  you  want  to  cover  more  than  one  site 
at  a  time,  you  will  need  help.  Sites  may  be  physically  distant,  like  two  sep¬ 
arate  training  areas.  There  may  be  a  lot  of  them  close  together,  like  a  number 
of  training  stations  in  one  training  area.  Or  it  may  be  that  more  than  one 
lesson  is  being  given  at  the  same  time  in  different  places.  The  point  is,  you 
can  only  be  at  one  place  at  a  time.  If  the  evaluation  requires  you  to  do  more 
than  this,  you  will  need  additional  personnel  to  act  as  training  observers. 

Not  everybody  makes  a  good  training  observer.  Soldiers  seem  to  sort  them¬ 
selves  out  into  three  different  categories  when  it  comes  to  observing  and  re¬ 
cording  training  data. 

There  are  those  who  just  cannot  seen  to  get  beyond  the  task  itself.  They 
get  lest  in  observing  how  the  task  is  performed.  They  either  already  know  how 
to  do  it  and  want  to  make  sure  that  the  instructor  is  teaching  it  correctly, 
or  they  do  not  know  how  to  do  it  and  get  caught  up  in  learning  how.  They  can¬ 
not  see  the  training  "process."  Subject  matter  experts  (SME)  quickly  fall  into 
this  pattern.  For  this  reason,  it  may  be  best  if  you  did  not  try  to  get  SMEs 
for  training  observers.  You  will  have  to  take  what  you  can  get,  of  course. 

Just  do  not  try  too  hard  for  SMEs.  Soldiers  who  have  trouble  "seeing"  training 
process  are  hard  to  identify.  You  may  be  able  to  Identify  them  from  their  per¬ 
formance  during  this  workshop. 

There  are  also  those  who  can  and  do  see  training  "process"  and  can  describe 
what  they  see  to  you  orally  but  have  a  lot  of  trouble  writing  down  what  they 
see.  The  problem  with  using  these  soldiers  is  that  you  will  not  get  a  record 
of  their  observations  from  them;  you  will  get  their  overall  impressions  and 
interpretations.  These  are  unique  to  them  and  not  very  useful  in  a  TPE.  They 
will  not  provide  you  with  data. 

Finally,  there  are  those  that  can  see  training  "process"  and  can  prepare 
useful  written  comments  on  what  they  see.  These  are  the  soldiers  you  will  want 
for  training  observers. 

One  purpose  for  the  workshop  is  to  sort  your  observers  into  these  groups 
as  best  you  can  before  they  begin  to  collect  real  data  for  you.  If  you  have  to 
keep  them  all,  you  can  use  the  workshop  experience  to  decide  who  will  do  what 
or  who  will  get  which  items  from  the  Observer's  Job  Aid  (SF  81-16). 

A  second  purpose  for  the  workshop  is  to  try  out  your  item  selection  from 
the  Observer's  Job  Aid.  It  will  give  you  a  chance  to  see  if  the  items  you  have 
chosen  answer  the  questions  you  have  about  the  program.  You  should  have  chosen 
some  difficult  items  (those  that  are  starred  in  RP  81-16)  and  some  easier  items. 
You  will  be  able  to  estimate  which  of  your  observers  will  be  more  able  to  pro¬ 
vide  data  on  the  more  difficult  items. 
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A  third  purpose  for  the  workshop  is  to  show  the  training  observers  that 
they  can,  in  fact,  see  training  "process"  and  make  useful  comments.  The  ex¬ 
perience  will  show  the  observers  that  the  worksheets  are  just  that;  sheets  for 
their  convenience  in  organizing  their  comments  and  helping  them  remember  what 
to  look  for.  The  workshop  should  reduce  their  anxiety.  You  can  use  it  to  tell 
them,  clearly,  that  the  worksheets  are  not  checklists  and  should  not  be  used 
as  checklists. 

A  fourth  purpose  for  the  workshop  is  to  force  you  and  your  observers  to 
try  the  system  one  time  before  you  go  live  with  the  real  training  program. 

The  descriptions  in  the  Observer's  Job  Aid  will  take  on  added  meaning  as  a  re¬ 
sult  of  the  workshop.  New  training  observers  often  have  a  "Yeah!  Sure!"  atti¬ 
tude  about  it  before  they  try  it  for  the  first  time.  After  they  try  it  once, 
they  may  find  out  that  it  requires  more  attention  and  effort  than  they  planned 
on  putting  into  it.  Some  will  provide  the  effort  and  some  will  not.  You  need 
to  be  able  to  tell  them  apart. 

The  main  purpose  of  the  workshop,  however,  is  to  train  the  observers  to 
use  the  Observer's  Job  Aid  and  the  worksheets  that  you  constructed  for  evalu¬ 
ating  the  training  program.  This  may  be  a  more  difficult  task  than  it  first 
appears.  The  following  "training  plan"  for  a  training  observer's  workshop  is 
provided  as  a  guide. 

1.  The  workshop  should  be  scheduled  for  three  consecutive  days,  i.e., 
Monday-Wednesday,  Tuesday-Thursday ,  or  Wednesday-Friday.  Try  not  to  break  the 
workshop  up  over  a  weekend.  Schedule  for  three  full  days.  You  may  not  use 
all  of  the  third  day  but  the  first  time  you  fall  to  schedule  all  of  it  will  be 
the  one  time  you  will  need  it.  Arrange  the  workshop  in  a  fashion  similar  to 
the  following. 

Day  1.  Morning.  Introduction  (objectives,  purpose,  schedule,  etc.). 

General  background  on  evaluation  (use  Sections  I 
and  II  of  this  Job  Aid  as  a  basis  for  this) . 

Afternoon.  Terminology  and  "demonstration"  (go  through  the 
items  that  will  be  on  Worksheets  2-4  in  detail, 
making  sure  that  they  understand  what  the  items 
mean,  what  they  need  to  "see,"  and  how  they 
should  record  their  observations).  Go  over  Work¬ 
sheet  1.  Use  your  presentation  as  a  sort  of 
demonstration  since  a  live  demonstration  will 
probably  not  be  possible.  If  you  can  get  a 
filmed  or  televised  piece  of  instruction,  you  may 
be  able  to  use  it  for  a  demonstration.  (See  RP 
81-16  for  Information  on  completing  the  worksheets) 

Day  2.  Morning  Prepare  to  observe  instruction  and  observe  in- 
nnd  struct  Ion.  The  lesson  to  be  observed  should  be 

Afternoon.  3  to  6  hours  long  and  should,  if  possible,  in¬ 
clude  a  test.  It  must  include  practice. 


Day  3.  Horning.  Each  obsarvar  organises  his  or  her  data. 

AlletlU'iw.  I.odtl  I  he  gi  •<!«)•  (it  gttltig  i>vei  ea>  ti  ><l<oet  vei  'a  .lata 

and  observations.  Summarize  the  experience  for 
the  group. 

2.  Try  to  have  no  aore  than  five  observers  in  the  workshop.  If  you 
have  aore  than  five,  you  will  have  a  hard  time  attending  closely  to  each  one. 
Since  one  of  the  reasons  for  having  the  workshop  Is  to  become  familiar  with 
the  potential  of  each  observer,  having  too  many  'to  do  this  well  defeats  that 
purpose.  In  addition,  more  than  six  observers  (five  plus  yourself)  In  a  train¬ 
ing  site  becomes  disruptive.  You  and  the  observers  will  be  a  distraction  in 
any  event.  Try  to  minimize  the  Impact  by  keeping  the  numbers  small. 

3.  Since  the  practice  part  of  the  workshop  will  Involve  observing 
actual  ongoing  training,  you  will  have  to  find  a  piece  of  training  to  observe. 
This  piece  of  training  can  be  part  of  a  formal  training  program  (institutional 
or  transition  training)  or  a  lesson  being  given  by  a  unit  as  part  of  a  unit 
training  program.  The  lesson  should  include  lecture,  demonstration,  and  prac¬ 
tice  events  and,  if  possible,  a  test.  At  least  one  week  before  the  workshop, 
arrange  for  the  group  to  observe  the  training.  Make  sure  the  primary  Instruc¬ 
tor  knows  the  group  Is  coming.  Get  the  lesson  plan  and  prepare  Worksheet  1 
(see  Section  IV  in  this  Job  Aid). 

4.  Provide  each  observer  with  a  copy  of  the  Observer's  Job  Aid  (BP 
81-16)  and  a  completed  Worksheet  1  at  least  24  hours  in  advance  of  the  workshop. 

5.  Have  the  group  meet  at  a  site  that  is  well  lighted,  free  of 
noise  distractions,  has  a  writing  surface  for  each  observer  (tables,  desks, 
etc.),  and  has  either  a  chalkboard,  large  tablet  and  easel,  or  butcher  paper 
and  easel.  You  will  need  this  site  for  day  1  and  day  3. 

6.  Provide  the  observers  with  the  training  objective  for  the  workshop 

TASK.  The  participant  will  observe  ongoing  training  and  testing 
so  as  to  answer  the  questions  on  Worksheets  2  (Training  Environment),  3  (Train¬ 
ing  Observation),  and  4  (Testing  Observation)  and  to  determine  if  the  training 
equlpment/materlals/alds,  job  aids,  training  site  requirements,  safety  pre¬ 
cautions,  and  training  events  listed  on  Worksheet  1  have  been  followed. 

CONDITIONS.  Training  observers  will  be  given  a  completed  Work¬ 
sheet  1  (Training  Plan),  blank  Worksheets  2-4,  and  a  stopwatch.  Observations 
will  take  place  at  any  assigned  training  site. 

STANDARDS.  For  each  item  on  Worksheets  2-4,  "yes"  or  "no"  is 
checked  to  Indicate  whether  the  actions  Implied  by  the  item  occurred  or  did 
not  occur.  Written  comments  are  made,  at  a  minimum,  for  each  "no"  recorded  on 
Worksheets  1-4.  Comments  are  factual  recordings  of  what  actually  happened. 
Opinions  are  clearly  labelled  as  such  and  are  not  substituted  for  factual  re¬ 
cordings.  Written  comments  are  legible. 


7.  Provide  the  observers  with  the  purpose  for  this  training,  i.e., 
to  train  than  to  use  the  Job  Aid  and  Worksheets,  to  give  them  dry-run  practice 
before  they  have  to  go  "wet,"  and  to  familiarize  you  with  their  aptitudes  so 
that  you  can  maximize  team  productivity  by  providing  differential  assignments, 
if  necessary. 

8.  Tell  the  observers  what  their  eventual  job  duties  will  be  so 
that  they  can  relate  this  training  to  those  duties.  Hake  sure  that  they  see 
that  the  workshop  is  really  a  practice  session  for  what  they  will  be  doing  as 
observers. 

9.  Training  observation  can  be  uncomfortable,  boring,  and  difficult. 
Personnel  selected  to  be  training  observers  will  need  some  motivating  to  get 
them  started  and  a  lot  of  attention  during  the  duration  of  their  assignment  to 
keep  them  going.  You  and  your  superiors  will  have  to  arrange  positive/ 
negative  consequences  for  learnlng/not  learning  (and  performing/not  performing) 
to  be  training  observers.  If  there  are  no  consequences,  the  observers  are 
likely  to  put  very  little  effort  into  the  task.  If  this  happens,  the  data  you 
get  from  them  will  probably  look  all  right  but  may  bear  little  relationship  to 
what  actually  occurred. 

10.  Provide  the  group  with  the  workshop  schedule. 

11.  Go  through  the  items  in  the  Observer's  Job  Aid  (or  the  items 
you  have  selected)  with  the  group  to  ensure  that  they  understand  what  it  is 
they  are  to  look  for,  how  they  will  know  when  it  happens  or  does  not  happen, 
and  how  they  should  record  their  observations.  Make  sure  that  all  the  ob¬ 
servers  have  the  same  interpretation  so  that  there  is  some  standardization 
across  observers.  Get  feedback  from  them  to  see  how  well  you  are  communicating 
and  to  ensure  understanding  and  commonality.  This  is  a  long  and  tiresome 
session.  You  may  want  to  break  it  up  by  giving  some  class  breaks.  If  you  do 
give  breaks,  give  them  at  logical  points  (between  Worksheets,  for  example) 

and  make  sure  you  tie  the  pre-break  activities  to  the  post -break  activities 
when  you  start  again. 

12.  After  the  long  session  in  11,  above,  you  may  consider  demon¬ 
strating  the  task  if  you  have  found  a  filmed  or  televised  instructional  se¬ 
quence.  Run  the  film  or  TV  tape,  commenting  on  what  you  see  that  relates  to 
the  items  on  the  Worksheets.  If  you  do  this,  one  or  more  rehearsals  will  be 
necessary.  You  will  be  much  more  effective  if  you  come  off  looking  like  an 
expert . 

13.  This  should  finish  day  1.  Make  sure  everyone  knows  what  is  go¬ 
ing  to  happen  on  day  2  and  where  it  is  going  to  happen,  what  they  need  to 
bring,  when  they  should  show  up,  what  the  uniform  requirement  is,  etc. 

14.  Day  2  should  be  used  to  actually  use  the  Worksheets  to  observe 
real  training.  All  of  the  observers  should  watch  the  same  training  so  they 
can  compare  their  observations  later  (day  3).  During  a  practice  event,  where 
more  than  one  training  station  is  operating  at  the  same  time,  they  should  split 
up  so  that  each  participant  has  one  station  that  he  or  she  observes  throughout 


the  practice  event.  Tell  the  observers  not  to  worry  about  getting  all  their 
comments  In  the  right  places  on  the  Worksheets.  This  will  come  with  practice. 
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wttuu  they  do  nut  know  where  to  put  their  coonueutu  during  iLuiuiug.  it  io  im¬ 
portant  to  capture  the  consent.  It  can  always  be  rewritten  in  the  right  place 
later.  Caution  your  observers  to  ovoid  interacting  with  either  the  soldiers 
or  the  instructors,  other  than  what  is  necessary  to  do  their  job.  They  should 
not  hide  anything  (their  notes,  their  purpose  for  being  there,  etc.)  but  they 
should  refer  all  questions  to  you.  The  teas  should  speak  through  one  voice; 
the  analyst' 8.  That  will  keep  the  normal  adversary  relationship  that  exists 
between  the  evaluated  and  the  evaluator  fros  becoming  too  much  of  a  problem. 

It  will  be  a  problem;  the  ides  is  to  keep  it  manageable. 

15.  If  the  group  does  a  lot  of  interacting  during  the  day,  with  each 
other  or  with  you,  keep  track  of  who  seams  to  he  relying  on  whom  so  that  you 
know  who  your  stronger  observers  are.  Observers  can  he  strong  in  two  senses; 
they  can  be  very  good  observers  or  they  can  he  very  persuasive.  Very  persua¬ 
sive  observers  will  only  be  of  value  if  they  are  also  very  good  observers. 

Poor  observers  who  are  persuasive  are  s  problem. 

16.  On  the  morning  of  day  3  have  each  observer  consolidate  his  or 
her  comments  onto  the  worksheets. 

17.  Go  over  each  item  on  each  Worksheet  (Part  III  of  Worksheet  1, 
Worksheets  2  and  3,  and  Worksheet  4,  if  used)  compering  the  comments  of  the 
various  members  of  the  group,  including  your  own.  Try  to  bring  the  group  to 
consensus  so  that  they  feel  that  the  final  conclusions  are  a  team  effort. 

You  will  learn  a  lot  about  team  members  during  the  struggle  for  consensus  that 
will  help  you  manage  the  team  later. 

18.  Collect  the  Worksheets  so  that  you  can  sec  for  yourself  what 
the  observers  actually  wrote  on  them.  You  will  find  that  soaie  observers  write 
a  great  deal.  Other  observers  write  very  little;  they  provide  their  interpre¬ 
tation  of  what  they  saw  or,  even  less,  they  talk  a  lot  about  what  they  saw  but 
do  not  write  much  of  this  down.  If  you  can  select  some  observers  and  release 
others,  select  those  who  write  a  great  deal  because  they  will  provide  the  most 
accurate  description  of  what  happened.  If  you  need  more  observers,  try  to 
avoid  those  who  Insist  on  giving  you  their  interpretation  (judgment)  of  what 
happened.  Pick  those  who  write  little  but  are  accurate  in  what  they  write. 

19.  That  will  complete  the  workshop  except  for  questions  and 


discussion 


SECTION  VI 


COLLECTING  DATA  DURING  PRACTICE  EVENTS 

Training  observers  are  data  collectors.  They  do  not  become  data  collec¬ 
tors  easily,  however.  They  have  to  be  trained.  Section  V  outlines  a  plan  for 
a  workshop  in  training  observation  and  data  collection.  This  section  supple¬ 
ments  Section  V  by  going  into  detail  on  how  to  collect  data  during  practice 
events. 

In  the  development  of  this  system  of  training  program  evaluation,  well 
over  one  hundred  training  observers  were  trained  during  pilot  runs  against 
actual  ongoing  training.  Observers  were  both  civilian  and  military.  The 
civilian  observers  were  in  grades  from  GS3  to  GS14.  The  military  observers 
were  in  grades  from  E4  to  05.  All  of  them  had  trouble  with  the  notion  of  col¬ 
lecting  "data"  during  practice  events.  The  natural  inclination  seems  to  be  to 
"watch"  what  is  going  on  without  recording  it  in  detail. 

The  problem  with  this,  of  course,  is  that  when  the  practice  event  is  over, 
all  the  observer  has  is  a  general  impression  of  what  happened.  Since  practice 
events  are  the  most  critical  events  for  training,  a  TPE  should  collect  more 
than  general  Impressions  about  how  these  events  were  conducted. 

The  analyst,  evaluator,  or  data  collection  team  chief  should  decide,  or 
at  least  coordinate  for  his/her  team,  which  types  of  data  to  collect  and  the 
minimum  depth  (detail)  of  that  data.  Observers  on  their  own  tend  to  count  the 
obvious  (e.g.,  hit/miss,  number  of  students  present),  but  then  record  addi¬ 
tional  data  which  are  not  recorded  by  the  other  observers.  Coordination  prior 
to  observation  maximizes  the  amount  of  useful  data. 

The  following  rather  lengthy  example  is  given  here  to  show  the  kinds  of 
data  available  during  practice  events  and  how  these  data  can  be  used  to  draw 
conclusions  about  the  training.  The  example  is  for  an  individual  training 
program  and  is  drawn  from  the  armor  Training  context.  Similar  data  also  can 
be  gathered  during  collective  training.  An  example  is  not  given  for  collective 
training,  however,  because  observational  constraints  vary  so  much  from  situa¬ 
tion  to  situation  that  a  useful  example  would  contain  so  much  situation  descrip¬ 
tion  in  it  that  no  one  would  want  to  wade  through  the  description  to  get  to 
the  example.  The  process  is  the  same;  simply  write  down  what  happened. 

Example 

Objective  1.  Task:  To  engage  and  score  a  "hit"  on  a  single  stationary 
target  using  a  ballistic  and  a  non-ballistic  reticle.  Condition:  Using  a 
Wiley  Burst-On-Target  Trainer.  Given  the  appropriate  fire  command.  Standard: 
Score  a  "hit"  on  each  of  two  separate  engagements  with  each  reticle.  Time 
limit  per  engagement  is  15  seconds. 

Objective  2.  Task:  To  engage  and  score  a  "hit"  on  a  single  moving  target 
using  a  ballistic  and  a  non-ballistic  reticle.  Condition:  Using  a  Wiley 
Burst-On-Target  Trainer.  Given  the  appropriate  fire  command.  Standard:  Score 
a  "hit"  on  each  of  two  separate  engagements  with  each  reticle.  Time  limit  per 
engagement  is  15  seconds. 
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Objective  3.  Task:  To  sense  and  adjust  fire  given  a  first  round  miss 
using  a  ballistic  and  a  non-ball is tic  reticle.  Condition:  Using  a  Wiley 
Burst-On-Target  Trainer  adjusted  to  produce  a  first  round  miss.  Given  the  ap¬ 
propriate  fire  commands.  Standard:  Make  the  correct  seising  and  score  a 
second  round  "hit”  on  each  of  two  separate  engagements  with  each  reticle.  Time 
limit  per  engagement  is  30  seconds. 

These  three  objectives  define  a  practice  event  wherein  each  soldier  should 
continue  in  practice  until  he  or  she  performs  to  standard.  Performance  to 
standard  is  required  on  each  of  12  separate  engagements. 


Ballistic  Reticle 


Stationary 

Target 

2  engagements 


Moving 

Target 

2  engagements 


Sense  and 
Adjust 

2  engagements 


Non-ballistic  Reticle 


2  engagements  2  engagements 


2  engagements 


Consider  four  separate  stations  each  containing  a  tank  gunnery  training 
device  (Wiley)  and  an  assistant  instructor,  a  class  size  that  mandates  six 
soldiers  per  station,  and  a  schedule  that  calls  for  two  50-minute  periods  for 
this  practice.  Observers  can  collect  the  following  Information  on  each  soldier 
at  each  station  (if  there  are  four  observers  -  one  for  each  station)  during 
the  PE: 

Number  of  engagements 
For  each  engagement: 

Engagement  type  (from  observations  and  instructor  comments) 

Time 

Hit  or  Miss 
Instructor  comments 

The  information  can  be  recorded  by  station  and  by  soldier  bn  a  blank  piece 
of  paper.  An  example  of  how  this  might  be  done  is  shown  in  Appendix  C. 


The  information  in  Appendix  C  tells  us  a  lot  about  the  practice  event  that 
would  not  otherwise  have  been  known.  (Without  this  information  only  global 
impressions  of  how  the  event  was  conducted  would  have  been  possible.  No  detail 
would  have  been  available  for  fixing  the  practice  event.)  From  the  data  in 
Appendix  C,  a  number  of  conclusions  about  the  practice  event  can  be  drawn. 

The  kinds  of  conclusions  that  can  be  reached  are  illustrated  below.  Notice 
that  the  information  presented  in  each  of  the  tables  was  taken  from  the  data 
in  Appendix  C. 

1.  The  practice  event  did  not  conform  to  the  lesson  plan.  Soldiers 
were  supposed  to  have  practiced  six  different  engagement  types.  A  station  by 
engagement  type  table  (Table  1)  shows  that  practice  with  the  non-ballistic 
reticle  was  virtually  Ignored.  Only  one  station  attempted  all  required  en¬ 
gagement  types. 
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Table  1 


Engagement  Types  Practiced  at  Each  Station 

Engagement  Type  Station 

12  3  4 

Non-ballistic  Stationary  x 

Non-ball 1st ic  Moving  x  x 

Non-balllstlc  Sense/Adjust  x 

Ballistic  Stationary  x  x 

Ballistic  Moving  x  x  x  x 

Ballistic  Sense/Adjust  x  x  x  x 

2.  There  was  no  uniformity  among  the  stations.  Each  assistant  In¬ 
structor  has  been  allowed  to  Ignore  the  requirements  of  the  lesson  plan  and 
conduct  practice  In  accordance  with  his  set  of  priorities.  This  defeats  the 
purpose  of  a  task  analysis,  criticality  judgments,  and  training/lesson  plans. 
"What  is  trained"  should  not  be  an  individual  Instructor  decision.  This  prac¬ 
tice  event  suggests  a  lack  of  supervision  by  training  management  (and  what  hap¬ 
pens  when  the  lesson  plan  is  Impossible  to  carry  out  given  time  and  resource 
constraints) . 

3.  Not  all  soldiers  practiced.  None  of  the  soldiers  practiced  until 
they  reached  the  required  performance  standards.  Table  2  shows,  by  station, 
the  number  of  soldiers  at  the  station,  the  number  of  soldiers  who  practiced, 
and  the  number  of  soldiers  who  practiced  until  they  achieved  the  training 
standards.  One-fourth  of  the  soldiers  had  no  opportunity  to  practice  the  tasks 
and  none  of  the  soldiers  achieved  the  standards  set  forth  in  the  lesson  plan. 

Table  2 

By  Station,  The  Number  of  Soldiers  at  the  Station, 

The  Number  of  Soldiers  Who  Practiced,  and  the  Number  of  Soldiers 
Who  Achieved  the  Training  Standard 

Station 

12  3  4 

No.  of  soldiers  6666 

No.  of  soldiers  who  practiced  6345 

No.  of  soldiers  who  achieved  0000 

t tic  training,  standard 


4.  Table  3  shows  the  results  for  the  140  engagements  conducted  dur¬ 
ing  the  practice  event.  A  number  of  things  can  be  said  about  the  practice 
event  from  Table  3. 

a.  Each  soldier  was  to  have  received  a  minimum  of  twelve  prac¬ 
tice  engagements.  Twelve  is  the  minimum  because  the  only  way  a  soldier  should 
have  gotten  just  twelve  was  to  have  met  the  standards  for  each  engagement  type 
on  the  first  try.  The  average  number  of  engagements  per  soldier  was  approxi¬ 
mately  six  (140/26  *  5.8).  Assuming  all  the  time  allotted  for  practice  was 
used,  this  shortfall  In  the  amount  of  practice  actually  conducted  suggests  the 
need  for  more  practice  time. 

b.  Most  of  the  practice  is  being  given  with  the  ballistic  reti¬ 
cle  (81Z).  This  may  be  realistic  if  the  ballistic  reticle  engagements  are 
harder  to  master.  The  data  from  this  practice  event  do  not  support  that  no¬ 
tion,  however.  This  imbalance  suggests  either  a  reassessment  of  the  require¬ 
ments  of  the  practice  event  or  on-site  supervision  by  the  principal  instructor 
to  ensure  compliance  with  the  lesson  plan. 

c.  The  time  standards  appear  unrealistic  for  this  stage  of 
training.  Only  4Z  of  the  engagements  met  the  time  standards. 

d.  The  practice  event  did  not  develop  the  skills  called  for  by 
the  lesson  plan.  Only  3Z  of  the  engagements  practiced  were  "successful,"  that 
is,  met  the  standards.  Soldiers  had  trouble  with  range  lines  on  the  ballistic 
reticle,  leading  targets,  and  adjusting  fire.  The  practice  event  clearly  did 
not  develop  the  skills  it  was  designed  to  develop. 

Table  3 

Results  of  the  Individual  Task  Training  Practice  Event 

Type  of  Engagement 


NB  Reticle 

B 

Reticle 

Sta 

Mov 

S/A 

Sta 

Mov 

S/A 

Totals 

ft  Engagements 
ft  Engagements: 

6 

15 

6 

18 

57 

38 

140 

Meeting  Time  Std 

0 

0 

1 

1 

0 

3 

5 

Below  Time  Std 

6 

15 

5 

17 

57 

35 

135 

Meeting  Hit  Std 

5 

7 

4 

7 

19 

18 

60 

Below  Hit  Std 

1 

8 

2 

11 

38 

20 

80 

"Successful"  Eng 

0 

0 

1 

1 

0 

2 

4 

"Unsuccessful"  Eng 

Reasons  for  Unsuccessful 

6 

15 

5 

17 

57 

36 

136 

Engagements: 

Aborted  by  Instr 

3 

3 

Wrong  Range 

11 

31 

42 

Wrong  Lead 

8 

32 

40 

Wrong  Sensing 

5 

5 

Wrong  Adjustment 

2 

20 

22 

Wrong  Sight  Picture 

1 

1 

5.  Note  that  the  "findings"  discussed  above  are  not  based  on  subjec¬ 
tive  impressions,  reconstructed  memory,  "expert"  opinion,  or  observer's  judg¬ 
ment.  The  "data"  is  objective;  the  number  of  engagements  were  counted,  times 
were  taken  with  a  stopwatch,  and  hlt/miss  and  reasons  for  "non-success"  were 
recorded  from  the  instructor's  comments  to  the  soldiers. 

6.  Three  things  should  be  evident. 

a.  This  is  what  happened.  Efforts  to  "explain  away"  these  re¬ 
sults  (a  poor  crop  of  soldiers,  bad  weather,  resource  constraints,  etc.)  will 
not  change  the  fact  that  the  practice  event  did  not  proceed  according  to  plan 
nor  attain  what  was  planned.  Untrained  soldiers  are  progressing  in  the  pro¬ 
gram.  They  are  probably  unprepared  for  what  is  to  cose.  The  question  now  is, 
"What  can  be  done  for  these  soldiers  to  bring  them  to  the  level  they  were  sup¬ 
posed  to  have  reached  during  this  event?"  The  answer,  in  the  institutional 
setting,  is  the  phantom  training  course  conducted  at  night  by  unit  cadre. 

b.  This  will  happen  again  and  again  unless  something  changes. 
This  "something"  may  be  the  Tasks,  Conditions,  and/or  Standards,  the  lesson 
plan,  or  the  Implementation  of  the  lesson  plan.  Some  combination  is  probably 
necessary. 


c.  This  "information"  would  not  have  been  available  had  the  ob¬ 
servers  not  recorded  what  they  saw.  If  they  had  used  checklists  (yes/no, 

OK/Not  OK,  GO/NO  GO) ,  or  had  relied  on  their  after-the-fact  expert  opinion, 
none  of  the  above  would  have  been  possible.  This  point  is  being  made  because 
most  observers,  without  training  and  supervised  practice,  do  not  know  how  to 
record  what  they  see.  The  natural  Inclination  is  to  watch  it,  and  then  attempt 
to  reconstruct  and  summarize  what  is  seen.  People  are  not  very  good  at  this. 
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SECTION  vri 


ANALYZING  TKST  BATA 

A  formal  evaluation  of  the  training  process  for  a  task  really  starts  when 
the  training  is  over  and  the  test  has  been  administered.  If  the  test  is  a  good 
measure  of  the  training  objectives,  the  scores  on  the  test  will  provide  a  lot 
of  Information  on  the  strong  and  weak  points  in  the  training.  Before  test 
scores  are  used  for  anything,  the  "goodness"  of  the  test  has  to  be  determined. 
This  involves  looking  at  four  aspects  of  the  test  Itself. 

1.  Was  the  test  administration  all  right?  Was  there  any  prompting 
during  the  test? 

2.  Was  the  task  that  was  tested  the  same  as  the  task  that  was 
taught?  Did  both  of  these  match  the  training  objectives? 

3.  Were  the  testing  conditions  more  or  less  difficult  than  the 
training  conditions?  Did  both  of  these  match  the  training  objectives? 

4.  Were  the  test  standards  higher  or  lower  than  the  training  stan¬ 
dards?  Did  both  of  these  match  the  training  objectives? 

Information  on  the  test  administration  will  come  from  Worksheet  4,  Testing 
Observation.  Key  items  are  numbers  90-93,  Test  Instructions,  and  103-106, 
Contamination,  from  RP  81-16,  the  Observer's  Job  Aid.  Contamination  is  the 
major  issue  here. 

If  the  soldiers  were  given  hints,  prompts ,  or  cues  during  testing,  the 
test  becomes  useless  as  a  measure  of  soldier  proficiency,  and  hence  the  test 
results  cannot  be  used  in  evaluating  the  effectiveness  of  the  training.  When¬ 
ever  test  contamination  is  present,  you  can  probably  safely  assume  that  the 
soldiers  cannot  perform  the  task  to  standard. 

Someone  may  suggest  that  the  soldiers  be  retested  without  the  hints, 
prompts,  or  cues.  That  will  not  help.  They  have  already  been  tested  and  will 
remember  much  of  what  is  on  the  test.  Retesting  will  give  artificially  high 
GO  rates. 

You  will  have  to  adjust  first  time  GO  rate  to  account  for  those  soldiers 
who  were  prompted  during  the  test.  Soldiers  who  received  a  GO  after  having 
been  prompted  should  really  be  considered  NO  GOs  for  TPE  purposes.  For  example: 

50  soldiers  tested 

45  soldiers  received  GOs 

5  received  NO  GOs  -  10%  official  NO  GO  rate  (minimum) 

But  10  of  the  45  soldiers  who  received  GOs  were  prompted 

Therefore,  the  actual  NO-GO  rate  may  be  (5+10) /50,  or  30% 

For  TPE  purposes,  then  the  first  time  NO  GO  rate  lies  somewhere  between  10%  and 
30%.  Ten  percent  is  the  lower  figure  because  that  is  the  official  NO  GO  rate. 
Thirty  percent  is  the  upper  figure  because  there  is  no  reason  to  believe  the 
10  other  soldiers  would  have  been  GOs  without  the  prompting.  For  TPE  purposes, 
use  the  30%  figure. 
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If  there  were  no  test  administration  problems  serious  enough  to  warrant 
questioning  the  usefulness  of  the  test  as  a  measure  of  soldier  performance  on 
the  tasks  just  taught,  then  look  to  see  if  a  performance  deficiency  exists. 

A  performance  deficiency  on  any  given  task  may  be  defined  by  the  first-time  MO 
GO  rate  on  that  task.  For  TPE  purposes,  a  performance  deficiency  is  said  to 
exist  whenever  the  first-time  NO  GO  rate  is  20%  or  greater. 

The  presence  of  a  performance  deficiency  may  indicate  that  the  training 
program  is  not  effective  in  teaching  the  required  skills.  But  a  performance 
deficiency  may  be  due  to  the  test  itself  or  result  from  a  mismatch  between  the 
objectives,  the  training,  and  the  test.  Similarly,  the  absence  of  performance 
deficiencies  does  not  necessarily  indicate  that  the  training  program  is  effec¬ 
tive;  the  absence  of  deficiencies  may  also  have  resulted  from  an  inadequate 
test  or  a  training-test  mismatch.  Evidence  of  problems  with  the  test  itself 
appears  on  Worksheet  4,  Testing  Observation,  and  solutions  to  each  problem  are 
specified  in  Section  VI  of  RP  81-17,  the  Modifications  Job  Aid.  Evidence  of  a 
training-test  mismatch  comes  from  comparing  information  appearing  on  Worksheets 
1,  3,  and  4.  Guidance  on  how  to  handle  the  various  kinds  of  training-test 
mismatches  is  provided  below.  Solutions  to  training-test  mismatches  are  also 
described  in  Section  VI  of  RP  81-17. 

The  task  that  was  tested  was  not  the  same  as  the  task  that  was  taught. 

The  idea  here  is  that  the  task  itself  was  different.  The  soldiers  were  asked 
to  do  something  in  a  different  manner  than  in  training.  For  example,  soldiers 
may  be  required  to  perform  a  task  during  training,  but  the  test  may  require 
soldiers  to  explain  how  the  task  is  performed  instead  of  actually  performing  it. 

It  is  important  that  the  procedures  for  completing  a  task  during  a  test 
be  the  same  as  the  procedures  practiced  during  training.  Some  may  argue  that 
the  test  is  where  the  soldier  is  expected  to  put  it  all  together,  so  the  exact 
procedures  will  differ  somewhat  from  those  taught  during  the  practice  sessions. 

Do  not  be  swayed  by  this.  As  long  as  new  material  or  new  procedures  or  new 
standards  or  new  conditions  are  being  introduced  to  the  soldier,  he  or  she  is 
being  trained.  When  training  is  over,  then  testing  starts.  Keep  training  and 
testing  clearly  separated  in  your  analysis.* 

Occasionally,  the  end  result  is  the  test  item  rather  than  the  procedures 
followed  to  get  to  it.  For  example,  when  soldiers  are  taught  to  disassemble 
the  M16  rifle  in  training,  the  focus  is  on  the  process.  The  standard  for  test¬ 
ing,  however,  may  be  a  rifle  that  has  been  disassembled  and  reassembled  in  a 
certain  time  period  without  particular  regard  to  the  process.  This  is  legiti¬ 
mate  as  long  as  the  soldiers  racticed  to  this  standard  during  the  final  stages 
of  training. 

The  remedy  here  is  to  ensure  that: 

If  the  process  is  being  tested,  it  is  the  same  process  as  was  practiced 

If  the  end  product  is  being  tested,  it  is  the  same  product  realized 
during  the  final  stages  of  training. 


In  your  analysis,  stick  to  what  you  know*  avoiding  the  temptation  to  go 
beyond  the  limits  of  your  data.  If  the  task  that  was  tested  was  not  the  same 
as  the  task  that  was  taught,  no  firm  statements  about  the  effectiveness  of  the 
training  program  can  be  made. 

The  task  was  not  taught.  Tasks  that  have  not  been  taught  should  not  be 
included  in  the  analysis.  The  purpose  of  a  training  program  evaluation  is  to 
evaluate  the  effectiveness  of  the  training  program,  not  the  soldier  or  the 
adequacy  of  the  front-end  analysis. 

Tasks  may  be  tested  but  not  taught  because  a  front-end  analysis  has  deter¬ 
mined  that  the  soldiers  should  be  proficient  in  the  task  from  previous  training 
and  experience.  The  test,  in  this  case,  will  measure  the  accuracy  of  that 
determination. 

Tasks  may  be  tested  but  not  taught  because  the  training  developer  wants 
to  get  a  check  on  proficiency  levels  for  future  training  decisions  or  baseline 
development.  That  is,  the  task  is  included  for  diagnostic  purposes. 

Tasks  not  taught  can  be  included  in  tests  for  many  reasons.  They  cannot 
be  used  to  evaluate  the  training  program. 

The  task  was  trained  but  not  tested.  Tasks  that  are  deemed  important 
enough  to  include  in  a  training  program  should  also  be  included  in  the  testing 
program. 

Tasks  that  are  not  tested  soon  lose  their  emphasis  in  training  and  gradu¬ 
ally  fade  out.  Since  the  job  of  the  trainer  in  criterion-referenced  instruc¬ 
tion  is  to  teach  to  the  test,  tasks  not  in  the  test  seldom  get  taught. 

Tasks  that  are  not  tested  should  not  be  included  in  the  analysis. 

The  test  conditions  were  more  difficult  than  the  training  conditions.  If 
there  was  no  performance  deficiency  (no  task  showed  more  than  a  20%  first  time 
NO  GO  rate),  then  the  soldiers  can  probably  perform  under  the  conditions  used 
in  training.  But,  which  conditions  are  correct?  Remember,  the  conditions  be¬ 
ing  considered  here  are  training  and  testing  conditions,  not  job  performance 
conditions.  Test  conditions  should  not  be  more  difficult  than  practice  condi¬ 
tions. 


If  there  was  a  performance  deficiency  (more  than  a  20%  first  time  NO  GO 
rate),  this  does  not  mean  that  the  soldiers  can  perform  to  standard  under  the 
training  conditions.  In  this  case,  nothing  can  be  said  about  the  effectiveness 
of  the  training  process.  The  soldiers  were  not  tested  under  the  training  con¬ 
ditions  so  you  do  not  know  if  the  program  met  its  goals  or  not. 

The  test  conditions  were  less  difficult  than  the  training  conditions.  If 
there  was  no  performance  deficiency,  then  the  soldiers  can  perform  to  standard 
under  these  less  difficult  conditions.  Nothing  can  be  said  about  performance 
under  the  more  difficult  conditions  used  in  training.  When  this  mismatch  occurs 
you  must  determine  which  conditions  are  correct,  and  modify  training  or  testing 
so  that  there  is  a  match  between  training  and  testing  conditions. 
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If  there  was  a  performance  deficiency  using  the  less  difficult  test  con¬ 
ditions,  the  soldiers  probably  cannot  perform  to  standard  under  the  training 
conditions  either.  Since  the  soldiers  were  trained  under  more  difficult  con¬ 
ditions  than  those  used  in  testing,  and  they  could  not  perform  to  standard 
under  the  easier  test  conditions,  the  training  process  probably  needs  to  be 
modified. 

The  test  standard  for  this  task  was  higher  than  the  training  standard. 

If  there  was  no  performance  deficiency,  the  soldiers  can  probably  perform  to 
the  training  standard,  and  the  training  for  this  task  was  probably  effective. 

If  there  was  a  performance  deficiency,  this  does  not  mean  that  the  soldiers 
can  perform  to  the  training  standard.  In  this  case,  nothing  can  be  said  about 
the  effectiveness  of  the  training  process.  The  soldiers  were  not  tested  against 
the  training  standard  so  you  do  not  know  if  the  program  met  its  goals  or  not. 

The  test  standard  for  this  task  was  lower  than  the  training  standard.  If 
there  was  no  performance  deficiency,  nothing  can  be  said  about  performance  at 
the  higher  training  standard.  If  there  was  a  performance  deficiency  with  a 
lower  test  standard,  the  soldiers  probably  cannot  perform  to  the  higher  train¬ 
ing  standard.  Since  the  soldiers'  test  performance  did  not  indicate  that  they 
could  perform  to  a  standard  lower  than  they  were  being  trained  on,  the  training 
process  probably  needs  to  be  modified. 

If  the  above  analyses  indicate  that  there  was  no  hinting,  prompting,  or 
cueing  going  on  and  that  the  test  task,  conditions,  and  standards,  matched  the 
training  (and  the  objectives)  task,  conditions,  and  standards,  then  the  result¬ 
ing  scores  can  be  used  in  determining  the  effectiveness  of  the  training  program. 

To  use  test  score  data  in  analyzing  the  effectiveness  of  training,  first 
obtain  the  actual  scoresheets  filled  out  by  the  examiners  during  the  end-of- 
block  tests.  Unless  there  is  an  urgent  need  for  the  original  scoresheets  else¬ 
where,  you  should  get  the  originals.  Copies  are  acceptable  if  they  are  made 
immediately  after  testing  is  over  and  before  they  have  been  reviewed  by  anyone. 
Be  aware  however  that  you  may  experience  some  delays  in  obtaining  copies  and 
the  copies  may  be  illegible.  Arrange  for  the  examiners  to  label  retests  so 
that  you  can  distinguish  first  test  information  from  retest  information. 

You  will  need  a  method  of  summarizing  the  data  obtained  from  the  score- 

sheets.  The  easiest  way  to  summarize  this  data  is  to  use  a  blank  copy  of  the 

actual  test  score  sheet.  You  can  write  whatever  test  identifying  information 
you  want  across  the  top  (unit  tested,  date,  etc.).  You  will  need  one  copy  of 
the  test  score  sheet  for  the  first  testing  and  one  for  each  retest  (first  re¬ 
test,  second  retest,  etc.)  that  you  want  to  keep  track  of.  Always  follow  at 

least  as  far  as  the  first  retest. 

Post  onto  these  blank  copies  of  the  test  score  sheet  the  number  of  soldiers 
tested,  the  number  receiving  GOs  and  NO  GOs,  and  the  percentage  receiving  NO 
GOs.  Figure  1  shows  a  blank  score  sheet  for  a  tracking  task.  Figures  2  and  3 
show  how  the  data  can  be  summarized  on  a  blank  score  sheet  for  the  original 
test  trial  and  the  first  retest. 


SCORE  SHEET 
TURRET  TRAINING  COURSE 
TRACKING  EXERCISE 


NAME _  RANK _  DATE 


Note:  The  soldier  will  position  himself  in  the  gunner's  seat  prior  to  beginning 
Che  test. 


START  TIME _  FINISH  TIME _ 

Requirement :  Track  shaded  portion  of  tracking  board 
using  the 


Did  the  soldier:  1.  Keep  the  aiming  point  on 

shaded  area  of  boards  #1 
and  #2  without  getting  off 
for  more  than  3  seconds? 

2.  Complete  this  requirement 
within  60  seconds? 


Standard:  Must  have  a  yes  on  both  items  to 
receive  a  GO. 


GPS  GAS 

yes  no  yes  no 


GO  NOGO  GO  NOGO 


Requirement:  Track  a  moving  target  using  the: 


Did  the  soldier:  1.  Track  the  moving  target 

without  getting  the  aiming 
point  off  the  target  for 
more  than  5  seconds? 

2.  Complete  this  requirement 
within  30  seconds? 


Standard:  Must  have  a  yes  on  both  items  to 
receive  a  GO. 


GPS  GAS 

yes  no  yes  no 


GO  NOGO  GO  NOGO 


Overall  Standard:  Must  have  a  GO  on  all  four  requirements  to  receive  an 

overall  GO. 

GO  NO  GO 


EVALUATOR 


SOLDIER 


Figure  1.  Blank  score  sheet  -  Tracking  Exercise 


SCOIvj-  SHEET 

TURRET  TRAINING  COURSE 
TRACKING  EXERCISE 


A  COMPANY 
TEST 


NAME  _ _ RANK _  DATE  15  Sep  81 

Note:  Tne  soldier  will  position  himself  in  the  gunner's  seat  prior  to  begir.  »ne 
the  tes..  SO  Soldiers  Tested 


Of  ARC  f  f  IE  0800 _  FINISH  TIME  1200 _ 

Reg q i r  .ant :  Track  shaded  portion  of  tracking  board 

using  the  GPS  GAC 


yes 

no 

yes 

HO 

Did  the  soldier.  1. 

Keep  the  aiming  point  on 
shaded  area  of  boards  #1 
and  #2  without  getting  off 

16% 

24% 

for  more  than  3  seconds? 

42 

8 

38 

12 

4. 

Complete  this  requirement 

47 

6% 

45 

10% 

within  60  seconds? 

3 

5 

GO 

NOGO 

GO 

NOGO 

St~irv~  ard ;  Must,  nave 

a  yes  on  both  items  to 

16% 

24% 

receive  a 

GO. 

42 

8 

38 

12 

Requite  ent:  Track  a  moving  target  using  the: 

GPS 

GAS 

yes 

no 

yes 

nc 

bid  thfc  ioldiir:  Track  the  moving  target 

without  getting  the  aiming 

50% 

58% 

pctni  off  the  target  for 

more  thai.  5  seconds? 

25 

25 

21 

29 

2..  Complete  this  requirement 

10% 

20% 

within  30  seconds? 

45 

JL 

40 

10 

GO 

NOGO 

GO 

N0G>' 

Standard:  livst  vave  a  yes  on  both  items  to 

50% 

58% 

receive  a  X. 

25 

25 

21 

29 

Overall  Standard.  Kut>l  have  a  GO  on  all  four  requirement!,  to  receive  un 

overall  GO. 


GO 

SO 


NO  60 

.  &L_ 


60% 


EwLUAIOR _ SOLDIER 

Note:  One  mem  failed  moving  target  with  GPS  and  passed  with  GAS  so  overall 
number  of  NO  GOs  is  29+1=30. 

Figure  2.  Completed  scoresheet  for  the  first  test  trial  -  Tracking  Exercise. 
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SCORE  SHEET 
TURRET  TRAINING  COURSE 
TRACKING  EXERCISE 


A  COMPANY 
1st  Retest 


VV'li  _ _ _  _ _ _ RANK  .  DATE  /V  *7 

c>.. :  V  ie  soldier  will  position  himself  in  the  gunner's  seat  prior  to  beginning 


STAR*  TIME  0800 _  FINISH  TIME  1200 


Requirement:  Track  shaded  portion  of  tracking  board 
using  the 


_  Overall  NO  GO 

Rate  after  1st 
Retest 

GPS  GAS 


yes  no  yes  no 

No.  of  Soldiers  Retested  —  8  12 


Jia  cae  soldier:  1.  Keep  the  aiming  point  on 


shaded  area  of  boards  #1 

2% 

6% 

and  #2  without  getting  off 

for  more  than  3  seconds? 

7 

J_ 

9 

_3_ 

2. 

Complete  this  requirement 

0 % 

2% 

within  60  seconds? 

8 

JL 

11 

_1_ 

GO 

NOGO 

GO 

NOGO 

Standard:  Must  have 

a  yes  on  both  Items  to 

2% 

6% 

receive  a 

GO. 

1 

_9_ 

3 

Requirement:  Track  a  moving  target  using  the: 

GPS 

GAS 

yes 

no 

yes 

no 

No.  of  Soldiers  Retested  — 

25 

29 

Did  the  soldier:  1.  Track  the  moving  target 

without  getting  the  aiming 

12% 

16% 

point  off  the  target  for 

more  than  5  seconds? 

19 

6_ 

21 

_8_ 

2.  Complete  this  requirement 

4% 

10% 

within  30  seconds? 

23 

2 

24 

5 

GO 

NOGO 

GO 

NOGO 

Standard:  Must  have  a  yes  on  both  items  to 

12% 

16% 

receive  a  GO. 

19_ 

6 

21 

8 

Q\  rail  Standard:  Must  have  a  GO  on  all  four  requirements  to  receive  an 

overall  GO. 

GO  NO  GO 

84%  16%  Overall  NO  GO  Rate  after  1st  Retest 

EVALUATOR  _  SOLDIER. _ ^ 


Figure  3.  Completed  scoresheet  for  the  first  retest  -  Tracking  Exercise 


From  the  summarized  data  in  Figure  2,  note  that  the  first  test  overall  MO 
GO  rate  is  60%.  We  will  assume  that  there  were  no  problems  with  the  test  it¬ 
self,  so  this  figure  is  useable.  This  high  MO  GO  rate  suggests  that  the  train¬ 
ing  for  this  task  was  not  effective.  The  goal  of  the  program  was  to  produce 
proficient  soldiers  on  the  four  subtasks.  It  clearly  did  not  do  that.  Only 
the  Tracking  Board/GPS  subtask  had  an  acceptable  first  time  NO  GO  rate  (i.e., 

20%  or  less). 

A  closer  look  at  Figure  2  indicates  that  the  soldiers  made  many  errors  in 
performing  the  tracking  tasks.  They  finished  in  time  but  their  error  rates 
were  too  high.  The  soldiers  had  more  trouble  with  the  GAS  than  with  the  GPS 
and  experienced  greater  difficulty  with  moving  targets  than  with  stationary 
targets. 

The  information  obtained  by  reviewing  the  data  summarized  on  the  score 
sheet  (Figure  2)  can  be  used  in  conjunction  with  other  information  about  the 
task  gathered  before  and  during  the  delivery  of  training  to  specify  changes  in 
the  training  given  for  that  task.  For  example,  data  from  Worksheet  3  may  in¬ 
dicate  that  the  soldiers  were  not  given  an  adequate  demonstration  in  tracking 
moving  targets  or  that  soldiers  did  not  practice  the  task  to  standard  when 
using  the  GAS.  Or  you  may  find  that  not  enough  time  was  allowed  for  all  sol¬ 
diers  to  practice  each  subtask.  Such  observation  leads  you  to  the  source  of 
the  performance  deficiency.  In  the  example  above,  the  large  number  of  errors 
made  in  using  the  GAS  may  have  resulted  from  soldiers  not  practicing  tracking 
with  the  GAS  to  standard  before  being  tested,  which  in  turn  may  have  resulted 
because  the  time  allotted  for  practicing  the  task  was  too  short.  Similarly 
the  lack  of  an  adequate  demonstration  may  have  adversely  affected  proficiency 
in  tracking  moving  targets. 

Analysis  of  the  test  score  data  can  identify  the  tasks  and  subtasks  for 
which  performance  deficiencies  exist  and  even  point  out  what  aspects  of  per¬ 
formance  (e.g.  time  or  procedural  requirements)  are  deficient.  But  test  data 
usually  cannot  be  used  to  identify  what  did  and  did  not  happen  during  training 
to  produce  the  deficient  performance.  To  find  the  causes  of  a  given  deficiency, 
you  must  carefully  review  the  data  recorded  on  Worksheets  1,  2,  and  3  during 
training  and  Worksheet  4  during  testing.  From  those  worksheets,  choose  the 
items  that  reflect  the  most  likely  causes  of  the  observed  deficiencies.  Then 
using  RP  81-17  recommend  changes  to  eliminate  the  cause  of  each  performance 
deficiency. 
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SECTION  VI n 

PROCEDURES  FOR  ANALYZING  AND  REPORTING  TRAINING 
AND  TESTING  PROCESS  DATA 

An  analysis  of  the  training  process  for  a  task  is  triggered  by  a  sequence 
of  events.  Each  of  the  events  (listed  below)  should  have  occurred  prior  to 
performing  the  analysis  of  the  training  process. 

1.  The  lesson  plan  has  been  evaluated  and  Worksheet  til  has  been  com¬ 
pleted  (see  Research  Product  (KP)  81-15  and  81-16). 

2.  The  training  has  been  observed  and  Worksheets  #2  and  if  3  have  been 
completed  (see  RP  81-16) . 

3.  The  test  has  been  observed  and  Worksheet  #4  has  been  completed 
(see  RP  81-16). 

4.  The  test  score  data  have  been  analyzed  and  no  problems  with  the 
test  or  test  administration  have  surfaced,  hence,  the  test  scores  are  con¬ 
sidered  to  be  useful  (see  Section  VII  in  this  job  aid). 

5.  The  first  time  NO  GO  rate,  for  TPE  purposes,  is  higher  than  the 
acceptable  minimum.  When  the  purpose  of  the  TPE  is  to  optimize  training  ef¬ 
fectiveness  and  efficiency  father  than  obtain  some  minimum  acceptable  profi¬ 
ciency  level,  the  NO  GO  rate  does  not  need  to  be  considered  as  a  condition  for 
performing  the  analysis. 

The  analysis  should  focus  on  the  following  seven  areas,  taking  one  at 
a  time. 

1.  Objectives 

2.  Training  Plan 

3.  Training  Site 

4.  Lecture  Events 

5.  Demonstration  Events 

6.  Practice  Events 

7.  Testing  Events 

Care  must  be  taken  to  say  something  about  each  area  and  to  report  the  good 
as  well  as  the  bad.  Remember  that  you  are  going  to  have  to  elicit  the  support 
of  the  training  developer  and  the  training  deliverer  in  changing  their  opera¬ 
tion.  They  will  resist  change.  If  all  you  present  is  the  bad,  they  will  ques¬ 
tion  your  objectivity  and  increase  their  resistance.  As  was  stated  earlier  in 
this  job  aid,  your  effectiveness  will  be  enhanced  if  you  are  part  of  an  evalu¬ 
ation  team  consisting  of  yourself  and  representatives  of  the  training  develop¬ 
ment  and  delivery  activities. 
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Your  job  at  this  point  is  to  take  all  the  information  available,  decide 
what  the  problem(s)  is,  and  recommend  specific  modifications  to  eliminate  the 
problem.  Research  Product  81-17,  A  Job  Aid  for  Modifying  Ineffective  or  In¬ 
efficient  Training  Programs,  will  help  in  this  process. 

Every  effort  should  be  made  to  correct  any  problems  at  the  worker  level. 
Individuals  at  the  directorate  and  staff  levels  need  to  be  kept  informed  about 
the  progress  of  the  evaluation  process  but  it  will  be  counter-productive  to 
inform  them  of  all  the  problems  identified.  They  should  only  be  asked  to  in¬ 
tervene  when  the  workers  are  at  an  impasse  and  need  someone  to  decide  in  favor 
of  one  action  over  another.  The  tendency  to  use  evaluation  data  to  rate  (or 
assess)  people,  rather  than  programs,  or  as  ammunition  to  force  a  decision,  is 
overwhelming  at  times.  It  is  much  easier  to  keep  the  data  from  being  misused 
in  this  way  if  it  is  not  released  beyond  the  evaluation  team.  Beware! 

Everyone  will  want  the  data  for  these  very  reasons.  If  they  get  it,  the  evalu¬ 
ation  program  will  rapidly  be  destroyed.  Your  data  may  need  command  protection 
at  the  highest  level. 

Analyzing  the  Objectives 

Evidence  for  problems  with  the  training  objectives  will  come  from  your 
lesson  plan  analysis  and  the  observer's  comments  on  Worksheet  1.  Several 
items  on  Worksheets  3  and  4  also  pertain  to  the  objectives  and  may  be  useful 
in  the  analysis. 

The  training  objectives  specify  the  intent  of  training  (what  the  training 
activity  Intends  to  accomplish;  what  the  expenditure  of  resources  is  supposed 
to  "buy"  for  the  Army).  The  objectives  should  be  clearly  stated  and  should 
describe  what  soldiers  will  be  able  to  do  after  training  that  they  could  not 
do  before  training,  the  conditions  under  which  they  will  be  able  to  do  it,  and 
the  standards  for  acceptable  performance.  Reasonable  agreement  on  these  ob¬ 
jectives  is  needed  between  the  training  manager,  developer,  and  deliverer.  If 
there  is  no  agreement  on  objectives,  evaluation  is  really  not  possible.  You 
will  not  be  able  to  get  a  handle  on  what  it  is  you  are  supposed  to  evaluate. 

Use  RP  81-15  and  Section  II,  RP  81-17  to  analyze  the  objectives.  See 
Section  IV,  this  Job  Aid,  for  instructions  on  filling  out  Worksheet  1  for  the 
observers.  If  an  educational  technologist  is  available,  have  him  or  her  help 
with  Worksheet  1. 
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EXAMPLE* 


Objective 


Source 


Task:  The  soldier  will  perform  operator 
maintenance  on  an  M240  machinegun. 
Condition:  One  M240  machinegun  per  two 
soldiers. 

Standard:  In  accordance  with 
TM  9-1005-313-10. 


Lesson  Plan  it 162509 
dated  1  Apr  79 


WORKSHEET  1,  TRAINING  PLAN  -  Part  III,  Training  Events 
Lesson  Title  Operator  Maintenance.  M240  Machinesun 


PR  AC  Di8a88embly 

PRAC  Assembly 


Training  stations  differed. 
At  some,  each  soldier  did 
both.  At  others ,  some  did 
assembly,  some  disassembly. 
No  consistent  standards. 


The  Task  statement  does  not  specify  what  the  soldier  will  do  following 
training  (see  question  #1,  RP  81-15).  "Perform  operator  maintenance"  appears 
to  be  specific  but  will,  in  fact,  permit  individual  instructor  interpretation 
during  the  conduct  of  training  (see  observer's  comments,  abov-s) .  Since  you, 
as  an  analyst,  probably  wrote  this  statement  on  Worksheet  1  as  part  of  your 
lesson  plan  analysis,  you  probably  had  trouble  trying  to  figure  out  just  what 
the  task  was  that  would  be  taught  from  this  lesson  plan.  If  the  lesson  plan 
had  spelled  out  the  task  in  detail,  you  would  have  spelled  it  out  in  detail  on 
Worksheet  1.  If  operator  maintenance  is  spelled  out  in  the  TM  and  those  pro¬ 
cedures  are  the  ones  that  are  going  to  be  taught,  then  the  appropriate  page 
and  section  numbers  can  be  used.  Something  has  to  be  done  to  make  this  more 
specific. 


*ln  the  examples  to  follow,  missing  sections  or  items  of  Worksheets  are  assumed 
to  indicate  "no  problems."  These  examples  are  overly  negative  so  that  a  number 
of  "problems"  can  be  illustrated.  Similarly,  the  report  following  the  examples 
is  more  negative  than  would  perhaps  be  usual. 


The  Conditions  statement  does  not  say  enough  (see  question  it 3,  RP  81-15). 
It  assumes  resource  constraints  that  are  unfortunate.  This  is  an  individual 
task;  sharing  a  machinegun  can  lead  to  "doer-watcher"  training  or  the  soldiers 
making  it  a  two  person  task.  No  mention  of  degraded  conditions  is  made  so  it 
has  to  be  assumed  that  this  task  will  not  have  to  be  performed  in  training 
while  masked.  That  does  not  appear  to  be  reasonable. 

The  Standard  is  not  a  standard;  it  references  the  TM  where  procedures  are 
spelled  out.  Standards  need  to  be  expressed  in  terms  of  accuracy  and  time  (see 
question  #4,  RP  81-15). 

Recommendation:  Either  spell  the  task  elements  out  or  reference  the  TM 
by  page  and  section.  Check  the  condition  to  be  sure  that  it  is  accurate. 

Spell  the  standard  out  in  terms  of  accuracy  (i.e.,  no  errors)  and  time  (i.e., 
four  minutes). 

Analyzing  the  Training  Plan 

Evidence  for  problems  with  the  training  plan  will  come  from  the  lesson 
plan  analysis  and  the  observer's  comments  on  Worksheets  1  and  3. 

The  training  (or  lesson)  plan  should  specify,  in  detail,  what  should  hap¬ 
pen  during  the  training  events.  This  analysis  can  be  conducted  on  either  of 
two  levels  depending  on  the  availability  of  an  educational  technologist.  If 
one  is  available,  turn  this  analysis  over  to  the  technologist.  Have  the  tech¬ 
nologist  a)  describe  the  learning  activities  in  the  lesson  plan  on  Worksheet 
1,  and  b)  evaluate  these  activities  as  builders  of  the  required  skills  and 
knowledges.  If  no  technologist  is  available,  do  a,  above,  yourself,  using 
Section  IV  of  this  job  aid  (and  Sections  III,  IV,  and  V  of  RP  81-17  for  back¬ 
ground).  You  may  also  try  b,  following  guidelines  provided  in  RP  81-15.  If 
the  nature  and  sequence  of  events  cannot  be  described  in  the  detail  required 
by  Section  IV  of  this  job  aid,  inform  the  training  developer  of  the  problem 
and  go  on  to  find  out  what  is  really  happening  on-the-ground  using  RP  81-16. 


EXAMPLE 


WORKSHEET  1,  TRAINING  PLAN  -  Part  III,  Training  Events 
Lesson  Title!  Operator  Maintenance,  M2 AO  Machlnegun 


TRAINING  EVENTS 


Task 

Egg 

Events 

1  Occur 

Comments 

No 

BBS 

ES3E3 

1 

TERM 

M240  terminology  given  to 
soldiers  (no  detail  in 
lesson  plan). 

/ 

Instructor  covered  barrel 
assy3  barrel  locking  latch , 
barrel  release 3  cover  assy, 
cover  latches 3  feed  tray 3 
flash  hider,  receiver  assy, 
receiver  gas  cylinder 3 
regulator  nut,  buffer  assy , 
trigger  and  frame  assy3 
charger  handle. 

1 

DEMO 

Demo  disassembly 

/ 

See  Worksheet  3. 

1 

DEMO 

Demo  assembly 

/ 

See  Worksheet  3. 

1 

PRAC 

Di8ae8embly 

/ 

Training  stations  differed. 
At  8ome3  each  soldier  did 

1 

PRAC 

Assembly 

/ 

both.  At  others,  some  did 
assembly ,  some  disassembly. 
So  consistent  standards. 

WORKSHEET  3,  TRAINING  OBSERVATION 


PART  II,  DEMONSTRATION/PRACTICE 


Observations 

Yes 

EZJ 

Comments 

38  -  Was  each  subtask 
demonstrated? 

✓ 

Missed  Task  1  -  assembly. 

Class  interrupted. 

The  terms  that  are  supposed  to  be  explained  to  the  soldiers  were  not  de¬ 
tailed  in  the  lesson  plan.  If  they  were,  they  should  have  been  listed  on  Work¬ 
sheet  1  so  the  observer  could  check  them  off  as  they  were  covered  (see  question 
t 7,  RP  81-15).  When  terms  and  concepts  are  not  specified  in  the  lesson  plan, 
this  frees  the  instructor  to  define  whatever  he/she  thinks  is  necessary.  This 
will  not  provide  standardization  across  instructional  events. 

The  lesson  plan  calls  for  demonstrations  but  does  not  spell  them  out  in 
enough  detail  to  ensure  that  each  demonstration  will  cover  the  essential  steps 
and  highlight  those  actions  that  are  difficult  or  novel  to  this  particular 
machinegun  (see  questions  #8  and  #9,  RP  81-15). 

42 


Note  that  the  observer  listed  the  terms  covered  during  the  TERM  event  (see 
Section  IV  of  this  job  aid  for  a  description  of  event  types).  Recommend  to 
the  training  developer  that  this  list  be  checked  against  TM  9-1005-313-10  and, 
if  the  list  is  accurate,  it  should  be  added  to  the  lesson  plan  for  the  guidance 
of  Instructors. 

Note  that  the  observer  indicated  that  "assembly"  was  not  demonstrated  and 
referred  to  Worksheet  3  for  why  it  did  not  occur.  More  data  on  why  "assembly" 
was  not  demonstrated  is  covered  below  under  Training  Site  and  Demonstration 
Events. 

Evaluating  the  Training  Site 

Evidence  for  problems  with  the  training  site  will  come  from  the  observer's 
comments  on  Worksheet  2. 

The  training  site  includes  everything  that  the  soldier  can  be  aware  of  in 
the  place  where  the  training  is  being  conducted.  This  includes  the  building 
and  its  contents  (if  any),  training  equipment  and  materials,  training  aids, 
light,  temperature,  etc.  (see  the  Training  Environment  items  in  RP  81-16). 
Criteria  against  which  to  evaluate  the  training  site  can  be  found  in  Section 
II,  Worksheet  2,  Training  Environment,  RP  81-16,  in  Table  1,  Training  Environ¬ 
ment,  RP  81-16,  and  in  RP  81-17  (Section  VI,  Training  Environment).  A  descrip¬ 
tion  of  the  training  environment  as  it  was  actually  encountered  on-the-ground 
during  training  will  be  supplied  by  the  training  observer  on  the  completed 
Worksheet  2.  Note  any  discrepancies  between  the  criteria  and  the  environment 
as  you  found  it  during  training.  The  decision  on  exactly  which  discrepancies 
to  highlight  for  the  various  interested  agencies  can  be  made  when  all  the  data 
are  in  and  your  report  Is  being  formulated. 

EXAMPLE 


WORKSHEET  2,  TRAINING  ENVIRONMENT 


Environmental  Factors 

Yes 

No 

NA 

1  -  Were  enough  instructors 

present  to  provide  adequate 
supervision  and  assistance? 

1 

/ 

1 

19-  Was  any  training  left  out  as 
the  result  of  an  interrup¬ 
tion? 

/ 

Comments 


Only  4  stations  operating. 
Soldiers  were  standing  around 
waiting. 


VIP  tour  came  in  during  demo. 
Instructor  missed  "assembly. " 
Not  enough  time  left  after  tour 
to  pick  it  up. 


Evidence  continues  to  accumulate  from  the  Worksheets.  Part  of  the  reason 
that  some  of  the  soldiers  did  not  practice  both  subtasks  may  be  that  there  were 
not  enough  assistant  instructors  available.  Recommend  that  the  training  de¬ 
veloper  consider  increasing  the  number  of  assistant  instructors  available  for 
this  practice  event  (see  Number  of  Instructors,  TRAINING  ENVIRONMENT,  Section 
VI,  RP  81-17). 
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The  observer  has  provided  the  reason  for  the  assembly  demonstration  being 
left  out  of  the  training.  This  is  an  Instance  of  a  training  problem  being 
caused  by  factors  beyond  the  control  of  the  training  Jevelop.-i  .uni  tin-  luutiu. 
tor  group.  When  such  instances  occur,  no  action  is  required  to  fix  the  train¬ 
ing  program.  Action  may  be  required,  however,  to  provide  extra  training  to 
offset  the  negative  effects  of  such  things  as  interruptions  on  the  soldiers  so 
affected.  Extra  training  may  be  necessary  for  this  group  of  soldiers  if  the 
missed  demonstration  and  practice  have  led  to  poor  soldier  performance  on  the 
end-of-block  test  (see  Distractions,  TRAINING  ENVIRONMENT,  Section  VI,  RP  81-17). 

Evaluating  Lecture  Events 

Evidence  for  problems  with  the  lecture  events  will  come  from  the  observer’s 
comments  on  Worksheet  1  and  from  Worksheet  3. 

A  lecture  event  is  any  training  event  where  the  primary  activity  is  the 
exchange  of  information.  These  events  are  sometimes  called  conferences  or  dis¬ 
cussions.  These  events  should  be  listed  in  Worksheet  1  from  the  lesson  plan 
analysis  or  should  be  recorded  in  the  comments  column  of  Worksheet  1  by  the 
training  observer  when  a  lecture  event  occurs  that  was  not  in  the  lesson  plan. 
Criteria  against  which  to  evaluate  lecture  events  can  be  found  in  Question  It 7, 
Section  IV,  RP  81-15;  TRAINING  OBSERVATION:  Terminology,  Table  2,  Section  II, 

RP  81-16;  and  LECTURE  EVENTS,  Section  VI,  RP  81-17.  Detail  on  what  actually 
happened  during  training  will  be  supplied  by  the  training  observer  on  the  com¬ 
pleted  Worksheet  3.  Discrepancies  between  what  should  have  happened  and  what 
actually  did  happen  (from  Worksheet  1)  and  between  the  criteria  and  what  actu¬ 
ally  did  happen  (Worksheet  3)  should  be  noted.  Worksheet  1  information  impacts 
on  the  implementation  question  (Was  the  training  Implemented  as  planned?)  and 
can  be  combined  with  other  implementation  information  for  a  separate  report 
if  desired.  Worksheet  3  information  can  be  used  to  identify  problems  and,  in 
conjunction  with  RP  81-17,  suggest  modifications. 

EXAMPLE 

WORKSHEET  1,  TRAINING  PLAN  -  Part  III,  Training  Events 


Lesson  Title  Operator  Maintenance,  M240  Machinegun 


Comments 


Instructor  covered  barrel 
assy,  barrel  locking  latch , 
barrel  release ,  cover  assy, 
cover  latches,  feed  tray, 
flash  hider,  receiver  assy, 
receiver  gas  cylinder, 
regulator  nut,  buffer  assy, 
trigger  and  frame  a83y. 


WORKSHEET  3,  TRAINING  OBSERVATION 
PART  I,  LECTURE/ CONFERENCE 


Observations 


Yes 


No 


NA 


Comments 


27  -  Were  soldiers  told  the 
training  objectives,  including 
tasks,  conditions,  and 
standards? 


Not  covered.  Instr  told  them 
that  he  was  going  to  teach  them 
to  assemble/disassemble  the  M240. 


28  -  Were  soldiers  told  why 
this  training  is  necessary? 


/ 


30  -  Were  soldiers  told  how 
this  training  fits  in  with 
previous  and  future  training? 


/ 


Did  not  connect  with  field  firing 
on  next  week's  training  schedule. 


36  -  Did  the  instructor 
provide  the  concepts  and 
terminology  needed  by  the 
soldiers? 


See  list  on  Worksheet  #1. 


From  the  observer's  comments,  you  know  that  the  soldiers  were  not  prepared 
for  this  block  of  instruction  by  the  instructor.  He/ she  did  not  tell  the  sol¬ 
diers  what  the  tasks  and  subtasks  were,  the  conditions  under  which  they  would 
have  to  perform  the  task  in  the  training  environment,  nor  the  standard  which 
would  be  applied  to  determine  if  they  would  get  a  "GO"  or  a  "NO  GO."  They  were 
not  told  why  they  were  being  trained  on  this  task  at  this  time  and  this  train¬ 
ing  was  not  put  into  perspective  with  respect  to  the  other  training  going  on. 
This  information  should  be  given  to  the  soldiers  at  the  beginning  of  the  block 
of  instruction  (see  Training  Objectives  and  Purpose,  LECTURE  EVENTS,  Section 
VI,  RP  81-17). 

During  the  examination  of  the  lesson  plan,  while  filling  out  Worksheet  1, 
you  would  have  determined  if  these  items  were  required  by  the  plan  and  if  the 
plan  contained  guidance  for  the  instructor.  If  these  items  were  not  required, 
recommend  that  they  be  added  to  the  lesson  plan.  If  these  items  were  required, 
recommend  to  the  training  manager  that  he/she  determine  the  reasons  behind 
this  oversight  so  that,  next  time,  these  items  will  be  covered. 

Evaluating  Demonstration  Events 

Evidence  for  problems  with  demonstration  events  will  come  from  the  ob¬ 
server's  comments  n  Worksheets  1  and  3. 
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A  demonstration  event  is  any  training  event  where  the  soldiers  are  being 
shown  how  to  perform  a  task  or  a  subtask.  These  events  should  he  listed  in 
Worksheet  1  from  the  lesson  plan  analysis  or  should  be  recorded  in  the  comment:! 
column  of  Worksheet  1  by  the  training  observer  when  a  demonstration  event  occurs 
that  was  not  in  the  lesson  plan.  Criteria  against  which  to  evaluate  demonstra¬ 
tion  events  can  be  found  in  RP  81-16  (Section  II,  Worksheet  3,  Training  Observa¬ 
tion,  and  in  Table  3,  Part  II,  Demonstration  and  Practice,  items  37-42)  and 
in  RP  81-17  (Section  VI,  Demonstration/Practice).  Detail  on  what  actually 
happened  during  training  will  be  supplied  by  the  training  observer  on  the  com¬ 
pleted  Worksheet  3.  Discrepancies  between  what  should  have  happened  and  what 
actually  did  happen  (from  Worksheet  1)  and  between  the  criteria  and  what 
actually  did  happen  (from  Worksheet  3)  should  be  noted.  Worksheet  1  informa¬ 
tion  impacts  on  the  implementation  question  and  can  be  combined  with  other  im¬ 
plementation  information  for  a  separate  report  if  desired.  Worksheet  3  informa¬ 
tion  can  be  used  to  identify  problems  and,  in  conjunction  with  RP  81-17,  to 
suggest  modifications. 


EXAMPLE 

WORKSHEET  1,  TRAINING  PLAN  -  Part  III,  Training  Events 
Lesson  Title  Operator  Maintenance,  M240  Machinegun 

TRAINING  EVENTS 


Task 

suns 

Events 

Occur 

Comments 

No 

Yes 

EE 

1 

DEMO 

Demo  disassembly 

/ 

See  Worksheet  3. 

1 

DEMO 

Demo  assembly 

/ 

See  Worksheet  3. 

WORKSHEET  3,  TRAINING  OBSERVATION 


PART  II,  DEMONSTRATION/PRACTICE 


Observations 


[Yes 


No 


NA 


Comments 


38  -  Was  each  subtask 
demonstrated? 


/ 


Missed  Task  1  -  assembly, 
interrupted. 


Class 


40  -  Could  demonstrations  be 
seen  and  heard  by  all  soldiers? 


/ 


I  couldn’t  hear.  Soldiers  in 
back  of  class  asking  each  other 
what  instr  said.  No  problem 
seeing  demo. 


41  -  Were  demonstrations  con¬ 
ducted  in  small  enough  steps  so 
that  the  soldiers  could  easily 
follow  the  instructor's  actions? 


/ 


Instr  went  through  immediate 
action  but  soldiers  did  not  follow 
very  well.  Six  questions  on  hand 
cycling  and  "safe”  during  demo. 

See  #50  on  this. 
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On  Worksheet  1,  the  observer  has  alerted  you  to  the  fact  that  assembly" 
was  not  covered  because  the  class  was  interrupted.  On  Worksheet  3  you  see  that 
some  of  the  soldiers  had  trouble  hearing  the  demonstration  of  "disassembly." 

The  observer  has  also  indicated  that  soldiers  questioned  the  "immediate  action" 
part  of  the  demonstration  (they  had  trouble  following  the  instructor).  The 
observer  also  alerts  you  to  other  problems  by  keying  you  to  item  50  which  is  a 
"Practice"  item  (this  is  covered  in  the  next  example,  below). 

Demonstrations  are  not  effective  for  those  soldiers  who  cannot  see  or  hear 
them  and  they  are  ineffective  when  soldiers  cannot  follow  and  understand  all 
the  steps  (see  Demonstrations,  DEMONSTRATION/PRACTICE,  Section  VI,  RP  81-17). 

Recommend  that  the  training  developer  look  at  this  demonstration  sequence 
again  to  ensure  that  it  is  as  effective  as  it  can  be.  The  evidence  from  this 
group  of  soldiers  suggests  that  it  needs  to  be  modified  to  eliminate  the  prob¬ 
lems  found  later  in  practice  (covered  in  the  next  example,  below).  If  the  in¬ 
structor  did  not  follow  the  lesson  plan  guidance  for  this  demonstration,  then 
additional  training  may  be  needed  to  sharpen  the  teaching  skills  of  the 
instructor . 


Evaluating  Practice  Events 

Evidence  for  problems  with  practice  events  will  come  from  the  observer's 
comments  on  Worksheets  1  and  3. 

The  importance  of  practice  in  training  cannot  be  overemphasized.  Practice 
is  where  skills  are  developed  and  brought  to  standard.  General  information  on 
practice  is  contained  in  Section  III,  RP  81-17,  and  a  separate  section  (Section 
VI)  in  this  job  aid  is  devoted  to  collecting  data  during  practice  events.  Most 
of  the  problems  in  training  are  with  practice  events.  It  is  not  uncommon  at 
all  to  find  insufficient  practice  (too  fev  soldiers  reach  standard),  instructor- 
specific  practice  (each  instructor  decides  for  himself  what  should  be  practiced 
and  to  what  standard),  and  even  no  practice  at  all  (walk-throughs  or  talk- 
throughs  only).  Practice  is  so  important  that  it  is  safe  to  say  that  tasks  not 
practiced  will  lot  be  learned  and  tasks  with  insufficient  practice  will  not  be 
retained  much  beyond  the  end  of  training  for  that  task.  Practice  events  should 
be  listed  in  Worksheet  1  from  the  lesson  plan  analysis  or  should  be  recorded  in 
the  comments  column  of  Worksheet  1  by  the  training  observer  when  a  practice 
event  occurs  that  was  not  in  the  lesson  plan.  Criteria  against  which  to  evalu¬ 
ate  practice  events  can  be  found  in  RP  81-16  (Section  II,  Worksheet  3,  Train¬ 
ing  Observation,  and  in  Table  3,  Part  II,  Demonstration  and  Practice,  items 
43-53)  and  in  RP  81-17  (Section  III  and  Section  VI,  Demonstration/Practice). 
Detail  on  what  actually  happened  during  training  will  be  supplied  by  the  train¬ 
ing  observer  on  the  completed  Worksheet  3.  Discrepancies  should  be  noted  as 
in  the  above  evaluations  of  lecture  and  demonstration  events.  Information  from 
Worksheets  1  and  3  will  have  the  same  uses  as  in  the  above  lecture  and  demon¬ 
stration  evaluations. 
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EXAMPLE 


WORKSHEET  1,  TRAINING  PLAN  -  Part  III,  Training  Events 
Lesson  Title  Operator  Maintenance,  M240  Machinegun 

TRAINING  EVENTS 


Task 

Events 

Occur 

Comments 

No 

■3T9I 

Yes 

No 

1 

PR  AC 

Disassembly 

/ 

Training  stations  differed. 

At  some ,  each  soldier  did 

1 

PRAC 

Assembly 

/ 

both.  At  others,  some  did 
assembly ,  some  did  disassem¬ 
bly.  No  consistent  standards. 

WORKSHEET  3,  TRAINING  OBSERVATION 
PART  III,  DEMONSTRATION/PRACTICE 


Observations 

Yes 

No 

NA 

Comments 

43  -  Did  all  soldiers  practice? 

J 

But  not  to  standard. 

47  -  Did  each  soldier  practice 
each  subtask  to  standard,  un¬ 
assisted,  before  going  on  to 
whole  task  practice? 

1 

/ 

27  soldiers  did  all  of  Task  1, 

12  soldiers  assembly  only,  19 
soldiers  disassembly  only.  See 
note  below. 

49  -  Were  the  soldiers  required 
to  meet  the  overall  standard  in 
performing  the  entire  task 
unassisted? 

/ 

Instructors  were  not  applying  a 
standard.  Each  soldier  did  it 
once  -  right  or  wrong. 

50  -  Did  a  lot  of  soldiers  have 
trouble  with  the  same  step 
during  practice? 

1 

16  did  not  hand  cycle  during  im¬ 
mediate  action.  42  forgot  to 
put  on  safe  when  pulling  charging 
handle  to  rear. 

55  -  Was  feedback  provided  as 
soon  as  possible  following 
actions? 

1 

/ 

Errors  pointed  out  when  subtask 
was  completed. 

57  -  Was  faulty  performance 
identified  and  corrected? 

■ 

/ 

II 

i 

Errors  pointed  out  but  no  second 
trials  were  allowed  so  soldiers 
could  get  it  right  one  time. 

PART  III,  GENERAL  OBSERVATIONS 

70  -  Were  the  soldiers  some¬ 
times  standing  around  with 
nothing  to  do  during  training? 

/ 

Only  4  stations.  Most  soldiers 
waiting  for  half  the  period  or 
more. 
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Practiced 

Note:  Station 

#  Soldiers 

Assembly 

Disassembly 

if  To  Standard 

1 

14 

1X1  INI 

xi  ixi  mi 

2 

IS 

1X1  INI 

LH1  W1  II 

mi  mi 

No  standards 
applied 

3 

14 

1X1  1X1  II 

4 

IS 

1X1  INI 

XI  XI 

From  the  lesson  plan  analysis  (Worksheet  1)  you  have  already  determined 
that  the  lesson  plan  does  not  specify  how  the  practice  events  are  to  be  con¬ 
ducted,  i.e.,  what  each  soldier  will  do,  the  subtask  and  task  standards  to  be 
reached,  the  conditions  that  need  to  be  set  up  for  each  soldier,  what  the  as¬ 
sistant  instructors  should  look  for  during  practice,  guidance  for  the  assistant 
instructors  in  how  to  remediate  particular  soldier  errors,  etc.  (see  questions 
//10-//13,  RP  81-15).  Remember,  if  this  Information  was  in  the  lesson  plan,  it 
should  have  been  outlined  for  the  observer(s)  on  Worksheet  1. 

You  also  have  additional  information  on  the  practice  event.  The  observer 
has  indicated  that  27  soldiers  practiced  both  assembly  and  disassembly  while 
31  only  practiced  half  the  task  (12  practiced  assembly  and  19  practiced  dis¬ 
assembly).  In  addition,  none  of  the  assistant  instructors  were  applying  a  con¬ 
sistent  standard  and  the  soldiers  were  not  being  required  to  perform  the  task 
to  standard,  each  soldier  getting  just  one  trial  whether  or  not  performance 
was  satisfactory. 

Regarding  feedback  from  instructor  to  soldier,  you  know  that  the  instruc¬ 
tors  were  not  stopping  soldiers  when  they  made  an  error.  They  were  waiting 
until  each  soldier  had  completed  a  subtask  before  telling  him/her  what  errors 
were  performed.  This  is  more  like  a  testing  technique  than  a  training  tech¬ 
nique  and  should  be  discouraged  (see  Section  IV,  FEEDBACK,  RP  81-17).  The  ob¬ 
server's  comments  in  item  57  indicate  that  soldiers  were  not  being  required  to 
perform  the  task  a  second  time  when  they  made  errors  the  first  time.  Soldiers 
should  be  required  to  correct  their  own  faulty  performance  before  going  on  in 
training  (see  Feedback,  DEMONSTRATION/PRACTICE,  Section  VI,  RP  81-17). 

The  observer  has  also  provided  some  diagnostic  information  in  items  41  and 
50  that  may  be  of  interest  to  the  training  developer.  Six  soldiers  had  prob¬ 
lems  with  "charging  handle"  and  "safe"  during  the  demonstration.  During  the 
practice  event,  16  soldiers  had  problems  with  the  "charging  handle"  and  43  with 
"safe."  The  training  developer  should  be  alerted  to  the  fact  that  this  part 
of  the  instruction  is  not  very  effective  (see  "Many  Soldiers  Had  Trouble  With 
The  Same  Step  During  Practice,"  DEMONSTRATION/PRACTICE,  Section  VI,  RP  81-17). 

Rivon  this  information  from  vour  lesson  plan  analysis  and  the  observer (s) 

.  ••nwtctii ,  and  ti-iiii*;  ::«•«•(  l**u  111,  riWiTliT,  KP  81-17  tor  background,  you  can 
make  the  above  recouuuendat ions  to  the  training  developer.  You  can  also  suggest 
that  this  lesson  be  revised  to  require  that  each  soldier  demonstrate  proficiency 
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and  to  include  a  prescription  (trainer  and  soldier  activities)  designed  to  ac¬ 
complish  proficiency.  Some  of  the  problems  with  this  practice  event  appear  to 
lie  with  inadequate  instructor  training  and  supervision  and  some  with  inadequate 
training  design.  Treat  the  design  problems  as  primary.  Good  training  design 
is  essential  and  the  harder  of  the  two  to  "fix"  up.  Instructor  training  and 
supervision  problems  are  likely  to  be  very  sensitive.  Tread  cautiously  in  this 
area  by  keeping  this  discussion  at  a  low  "worker"  level. 

Evaluation  of  Testing  Events 

Evidence  for  problems  with  testing  events  will  come  from  the  observer's 
comments  on  Worksheets  1  and  4. 

There  are  three  kinds  of  tests  to  be  concerned  with  here.  The  end-of- 
block  test  is  given  almost  immediately  after  training  on  a  task  (or  related  set 
of  tasks)  is  over.  Usually  there  are  24  hours  or  less  between  the  end  of  train¬ 
ing  and  the  test.  This  kind  of  test  contains  the  most  information  for  TPE 
purposes.  It  tells  the  soldiers  and  the  trainers  what  skills  and  knowledges 
the  soldiers  have  learned  and  it  tells  the  trainers  how  successful  the  training 
program  has  been  in  achieving  its  objectives.  This  kind  of  test  contains  the 
fewest  number  of  uncontrolled  factors  that  will  intervene  to  cloud  the  direct 
relationship  between  training  and  proficiency.  Tests  covering  several  blocks 
of  instruction  given  periodically  throughout  a  course  will  tell  the  soldiers 
and  the  trainers  what  proficiencies  the  soldiers  have  but  will  not  control  for 
non-program  effects  (outside  practice,  other  training,  peer  instruction). 

These  tests  are  useful  for  TPE  purposes  if  the  extent  to  which  these  non-program 
effects  occurred  can  be  estimated.  Since  these  estimates  can  seldom  be  made 
with  any  rigor,  these  tests  are  of  limited  usefulness  to  TPE.  End-of-course 
tests  are  usually  given  for  the  purpose  of  certifying  soldiers.  The  notion 
here  is  that  one  has  to  be  proficient  in  certain  tasks  to  receive  a  graduation 
certificate,  award  of  an  MOS,  additional  skill  identifier,  etc.  These  tests 
have  the  least  usefulness  from  a  TPE  standpoint.  Non-program  effects  are  at  a 
maximum  in  this  kind  of  test. 


You  will,  of  course,  have  to  use  what  the  training  plan  calls  for  in  the 
way  of  testing  events.  Some  will  argue  that  the  end-of-course  test  is  the  only 
true  measure  of  training  program  effectiveness.  In  a  sense  they  are  rigjit. 

If  the  program  does  not  produce  trained  soldiers,  it  is  not  effective.  The 
evaluator,  however,  must  ask,  "What  program  produced  these  results?"  If  the 
program  goals  are  not  clearly  specified,  with  each  training  event  being  clearly 
designed  to  bring  soldiers  to  each  enabling  objective  (in  sequence)  and  termi¬ 
nal  objective,  and  with  the  training  events  following  the  training  design  each 
time  the  event  is  given  (no  phantom  training  —  outside  practice,  peer  instruc¬ 
tion,  etc.),  the  evaluator  can  only  argue  that  what  the  program  produced  this 
time  may  not  be  what  it  will  produce  next  time.  The  system  is  unstable  and 
will  produce  unstable  results.  Knowing  that  a  program  has  produced  an  accept¬ 
able  (or  unacceptable)  product  is  only  of  use  to  a  training  manager  if  he  or 
she  is  reasonably  certain  that  it  is  likely  to  happen  again  the  next  time  the 
training  program  is  given. 


Testing  events  should  be  listed  in  Worksheet  1  from  the  lesson  plan  analy¬ 
sis  or  should  be  recorded  in  the  comments  column  of  Worksheet  1  by  the  training 
observer  when  a  testing  event  occurs  that  was  not  in  the  lesson  plan.  Criteria 
against  which  to  evaluate  testing  events  can  be  found  in  RP  81-16  (Section  II, 
Testing  Observation,  and  in  Table  5,  Testing  Observation,  items  90-109)  and  in 
RP  81-17  (Section  VI,  Testing).  Detail  on  what  actually  happened  during  testing 
will  be  supplied  by  the  training  observer  on  the  completed  Worksheet  4. 

See  Section  VII  in  this  job  aid  for  guidance  in  analyzing  test  scores. 

EXAMPLE 

BASIC  ARMOR  TEST 


,  NO  GO '8  added  - 

See  # 106  an 

.  Worksheet  4 

GO 

STATION  #1  M240  MACHINEGUN 

REQ  1 

Clear  M240  +12  NO  GO 

0  38 

r 

REQ  2 

Disassemble 

53 

.  n 

REQ  3 

Reassemble  +  8  NO  GO 

0  41 

n 

REQ  4 

Function  Check  +  12  NO  GO 

0  39 

t 

REQ  5 

Load  the  M240 

57 

• 

REQ  6 

Immediate  Action  +  11  NO  GO 

0  36 

■  m 

■  m 

NO  GO 


1st  2d  3d  4th 


REMARKS 


NO  GO  Rates 
Official  Adjusted 


16% 

29% 

12% 

33% 

2% 

2% 

19% 

38% 

51 


WORKSHEET  4,  TESTING  OBSERVATION 


Observations 


Yes 


No 


NA 


Comments 


93  -  Were  pass/fail  standards 
clearly  explained  such  that 
soldiers  would  know  when  they 
performed  correctly? 


94  -  Did  the  test  occur  soon 
after  the  completion  of 
training? 


98  -  Were  the  standards 
specified  in  the  training 
objectives  used  to  score 
test  performance? 


104  -  Did  the  examiner  help 
the  soldiers  in  any  way 
during  the  test? 


/ 


Read  from  test  instruction  sheet. 
No  questions  from  soldiers. 


/ 


Immediately  after  final  practice 
event. 


/ 


Examiners  did  not  have  stop¬ 
watches  and  tested  two  soldiers 
at  a  time.  Time  standards  not 
rigid.  See  #104,  #105,  #106. 


#  Soldiers  over  time  but  given  GO 


REQ  ST A  1 

1 
2 

3  -  2  - 

4  -  1 

5 

6  -  2  ~ 


STA  2 


105  -  Did  the  examiners  apply 
the  standards  specified  in 
the  training  objectives  con¬ 
sistently  and  objectively? 


106  -  Did  the  soldiers  re¬ 
ceive  artificial  cues  or  help 
during  the  test? 


✓ 


#  Soldiers  given  cues 


REQ  STA  1 

1 - 10  - 

2 

3  -  4  - 

4  -  8  - 

5 

6  -  6  - 


STA  2 
•  2 

■  1 
■  3 

•  2 


The  official  number  of  first  time  NO  GOs  should  be  listed  on  the  test 
score  sheet  as  in  the  example  above.  These  are  the  numbers  to  the  left  under 
..the  ’’GO"  and  "1st  NO  GO"  columns.  The  official  percentage  of  the  soldiers  who 
were  first  time  NO  GO  is  recorded  in  the  REMARKS  column.  Note  that  all  of 
these  percentages  are  less  than  20%.  This  suggests  that  there  is  little  need 
for  a  rigorous  analysis  of  the  M240  training  block.  However,  the  observer(s) 
has  reported  some  problems  with  test  administration  on  Worksheet  4.  Some  sol¬ 
diers  who  received  "GOs"  were  given  cues  during  testing  and  others  were  not 
required  to  meet  time  standards  (note  that  instructors  did  not  have  stopwatches) 
The  observer  has  provided  counts  of  these  in  items  #104  and  #106.  Note  that 
two  soldiers  exceeded  the  time  limit  on  REQ  3  at  Station  1,  etc.  Also,  ten 
soldiers  were  given  cues  during  REQ  1  at  Station  1,  etc.  The  soldiers  who  ex¬ 
ceeded  the  time  limits  and  the  soldiers  who  were  given  cues  are  considered  NO 
GOs  for  TPE  purposes  (see  Section  VII  in  this  job  aid) .  These  soldiers  have 
been  added  to  the  official  NO  GO  counts  on  the  test  score  sheet,  e.g.,  the  12 
soldiers  who  were  given  cues  on  REQ  1  but  were  considered  "GO"  officially  have 
been  added  to  the  NO  GO  count  for  REQ  1.  This  addition  has  increased  the  "NO 
GO"  percentage  of  14%  (official)  to 34%  (adjusted).  Similar  adjustments  have  been 
made  in  the  data  for  REQ  3,  4,  and  6.  Now  it  appears  that  an  analysis  of  this 
block  is  warranted  since  four  of  the  six  subtasks  had  adjusted  first  time  NO 
GO  rates  of  20%  or  more. 

Reporting  the  Results  of  the  TPE  to  Training  Developers/Deliverers. 

When  reporting  the  results  of  a  TPE  to  the  training  developer  and  de¬ 
liverer,  remember  to  cover  each  of  the  seven  items  at  the  beginning  of  this 
section.  Two  other  very  important  things  to  remember  are  to  balance  the  report 
with  positive,  as  well  as  the  negative,  comments,  and  to  focus  on  those  few 
negative  comments  that  will  have  the  greatest  impact  on  the  training  program. 

You  can  start  by  making  a  list  of  the  things  you  might  say  (the  example,  above, 
is  continued  here) . 


Objectives: 


Training  Plan: 


not  spelled  out. 


Positive 

Lead  to  performance 
oriented  training. 

Are  actually  spelled  out 
in  the  body  of  the  lesson 
plan  (but  are  hard  to 
follow  in  that  form) . 

Follows  the  usually 
accepted  model  for 
training  (lecture,  demon¬ 
stration,  practice,  test). 


Negative 

Task  statement  does  not 
specify  what  the  soldier 
will  do  following  training. 

Condition  statement  does  not 
say  enough. 

Standard  is  not  a  standard. 

Required  terms  are  not  de¬ 
tailed  in  the  lesson  plan. 

Demonstration  procedures  are 


Procedures  for  conducting 
practice  events  are  not 
spelled  out. 


Positive  Negat ive 

training  Sice;  Training  equipment.  Nut  enough  aeatoLaut  imuu. 

materials,  lighting,  tors  available, 

noise,  etc.,  were  all 

right  (no  negative  com-  Class  interrupted, 
ments  by  observer). 

Lecture  Events:  Terms  needed  by  soldiers  Objectives  and  reason  for 

given  to  them  by  instruc-  training  not  given  to  sol- 

tor.  diers. 

Demonstration  Events:  Some  soldiers  had  trouble 

hearing. 

Soldiers  had  trouble  follow¬ 
ing  "immediate  action." 

"Hand  cycling"  and  "safe"  led 
to  questions. 

All  soldiers  did  not  practice 
all  subtasks. 

Soldiers  were  not  required  to 
perform  to  standard. 

Feedback  to  soldiers  delayed 
to  end  of  practice  trial. 

Soldiers  had  problems  with 
"charging  handle"  and  "safe." 

Testing  Events:  Instructions  read  to  sol-  Standards  not  enforced. 

diers  from  lesson  plan. 

Some  soldiers  were  given  cues 
Pass/fail  standards  read  during  testing, 
to  soldiers. 

All  subtasks  were  tested. 

An  examination  of  the  list  above  suggests  that  the  major  problem  with  this 
block  of  instruction  is  the  training  prescription;  the  objectives  are  not 
clearly  spelled  out,  required  terms  are  not  detailed,  and  procedures  for  con¬ 
ducting  the  demonstration  and  practice  events  are  not  spelled  out.  Instructors 
are  training  from  a  lesson  plan  that  does  not  provide  much  guidance.  If  the 
lesson  plan  is  modified  to  add  this  guidance  via  a  training  prescription,  prob¬ 
lems  with  the  lecture,  demonstration,  and  practice  events  may  clear  up.  In  any 
event,  criticizing  the  instructors  at  this  point  would  be  counterproductive. 

If,  after  the  lesson  plan  has  been  modified,  these  problems  persist,  then 
training  management  should  be  alerted  to  the  need  for  additional  instructor 
training  and  supervision. 


Practice  Events:  All  soldiers  practiced 

(no  walk-throughs  or 
talk-throughs  substituted 
for  hands-on  practice). 
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In  addition,  there  are  two  problems  with  test  administration;  standards 
are  not  being  enforced  and  examiners  are  providing  cues  to  soldiers.  Recommend 
that  examiners  receive  additional  training  and  supervision  in  applying  standards 
consistently  and  avoiding  cueing. 

From  this  example,  then,  a  report  like  the  following  could  be  prepared  for 
the  training  developer  and  deliverer.  An  abbreviated  version,  leaving  out  much 
of  the  detail,  could  be  prepared  for  the  command  element. 

OBJECTIVES.*  The  task  statement  does  not  specify  what  the  soldier  will  do 
following  training.  Current  wording  does  not  provide  clear  guidance  regarding 
the  goal  of  this  training  (lesson  plan  analysis,  questions  #1  and  1t2,  RP  81-15). 
The  conditions  statement  does  not  clearly  spell  out  the  conditions  under  which 
the  task  must  be  performed  (lesson  plan  analysis,  question  it 3,  RP  81-15).  The 
standard  does  not  provide  the  instructional  staff  with  immediate  guidance  re¬ 
garding  what  constitutes  a  "GO"  and  a  "NO  GO"  (lesson  plan  analysis,  questions 
#4-#6,  RP  81-15).  Recommend  that  the  training  objective  be  rewritten  following 
the  guidance  in  Section  II,  RP  81-17. 

TRAINING  PLAN.  The  terms  needed  by  the  soldiers  for  performance  of  this 
task  have  been  left  out  of  the  lesson  plan  (lesson  plan  analysis,  question  it 7, 

RP  81-15) .  The  plan  calls  for  each  subtask  to  be  demonstrated  but  does  not 
provide  guidance  for  these  demonstrations  that  covers  the  essential  steps  and 
alerts  the  instructors  to  difficult  or  novel  actions  that  should  be  emphasized 
(lesson  plan  analysis,  questions  #8  and  99,  RP  81-15).  Practice  events  are 
required  by  the  lesson  plan  for  each  subtask  but  detailed  guidance  for  instruc¬ 
tors  on  how  to  conduct  these  events,  what  soldier  errors  to  look  for  during 
practice,  and  how  to  correct  faulty  performance,  is  missing  (lesson  plan  analy¬ 
sis,  questions  //10-//13,  RP  81-15).  Recommend  that  a  detailed  training  prescrip¬ 
tion  be  written  for  this  training  that  spells  out,  in  detail,  procedures  for 
conducting  the  lecture,  demonstration,  and  practice  events.  Guidance  for  this 
can  be  found  in  Section  IV,  RP  81-15,  and  Sections  II-VII,  RP  81-17. 

TRAINING  SITE.  Problems  with  the  training  site  were  minimal.  There  was 
enough  training  equipment  to  go  around  (item  4,  RP  81-16),  all  training  equip¬ 
ment  worked  properly  (item  5,  RP  81-16) ,  the  physical  layout  of  the  training 
site  made  It  easy  to  see  the  demonstrations  (item  10,  RP  81-16) ,  the  site  was 
reasonably  quiet  (items  12  and  13,  RP  81-16),  there  was  enough  light  for  the 
soldiers  to  see  what  was  going  on  (item  14,  RP  81-16),  and  the  weather  was 
reasonable  (items  21  and  22,  RP  81-16).  Soldiers  were  observed  standing  around 
waiting  during  practice  (item  1,  Worksheet  2),  however,  which  suggests  the  need 
for  additional  instructors.  This  problem  may  yield  to  a  revised  training  plan 
that  specifies  the  procedures  for  conducting  the  practice  event.  See  Number 
of  Instructors,  TRAINING  ENVIRONMENT,  Section  VI,  RP  81-17. 

The  "assembly"  subtask  demonstration  was  left  out  of  training  because  of 
a  VIP  tour  interruption  (item  19,  Worksheet  2).  Nineteen  soldiers  did  not  get 
a  chance  to  practice  "assembly"  (item  47,  Worksheet  2)  and  seventeen  soldiers 


information  in  parentheses  would  not  be  reported.  It  is  included  so  that  you 
can  see  the  basis  for  the  comment. 


(29%  of  this  group)  were  either  a  "NO  GO"  on  the  test  (test  score  sheet),  were 
given  a  "GO"  despite  their  exceeding  the  time  standard  (item  104,  Worksheet  4), 
or  were  given  cues  during  the  requirement  (item  106,  Worksheet  4).  Remedial 
training  for  this  group  on  this  subtask  may  be  beneficial.  See  Distractions, 
TRAINING  ENVIRONMENT,  Section  VI,  RP  81-17. 

LECTURE  EVENTS.  Instructors  followed  the  lesson  plan  (item  63,  Worksheet 
3)  as  best  they  could  given  the  vague  prescription  for  these  events.  The  pri¬ 
mary  instructor  filled  in  where  the  lesson  plan  was  vague  and  provided  the 
soldiers  with  the  terminology  from  TM  9-1005-313-10  (Worksheet  1) .  The  primary 
instructor  did  not  provide  the  soldiers  with  a  clear  description  of  the  train¬ 
ing  objective  (item  27,  Worksheet  3),  did  not  indicate  why  this  training  is 
necessary  now  (item  28,  Worksheet  3),  and  did  not  indicate  how  this  training 
fits  in  with  previous  and  future  training  (item  30,  Worksheet  3).  These  prob¬ 
lems  may  yield  to  a  revised  training  plan  that  specifies  the  procedures  for 
conducting  the  lecture  event.  See  Training  Objectives  and  Purpose,  LECTURE 
EVENTS,  Section  VI,  RP  81-17. 

DEMONSTRATION  EVENTS.  All  subtasks  were  demonstrated  with  the  exception 
of  the  "assembly"  subtask  (item  38,  Worksheet  3).  Some  soldiers  had  difficulty 
hearing  the  demonstrations  (item  40,  Worksheet  3).  Instructors  could  usefully 
be  reminded  to  ensure  that  all  soldiers  can  hear  adequately.  Some  soldiers  had 
trouble  following  "hand  cycling"  and  "safe"  during  the  demonstrations  (item 
41,  Worksheet  3).  These  problems  may  yield  to  a  revised  training  plan  that 
specifies  the  procedures  for  conducting  the  demonstration  events.  See  Demon¬ 
strations,  DEMONSTRATION/PRACTICE,  Section  VI,  RP  81-17. 

PRACTICE  EVENTS.  All  soldiers  received  hands-on  practice  (item  43.  Work¬ 
sheet  3).  Twenty-seven  soldiers  practiced  all  the  subtasks,  12  practiced 
"assembly"  only,  and  19  practiced  "disassembly"  only  (item  47,  Worksheet  3). 

No  data  is  available  on  the  extent  to  which  soldiers  were  reaching  the  standard 
since  the  instructors  were  not  applying  the  standard  during  practice  (item  49, 
Worksheet  3).  Two  steps  in  performing  this  task  proved  difficult  for  a  number 
of  soldiers.  Sixteen  did  not  "hand  cycle"  during  immediate  action  and  43  for¬ 
got  to  put  the  weapon  on  safe  when  pulling  the  charging  handle  to  the  rear 
(item  50,  Worksheet  3) . 

Instructors  were  pointing  out  soldier  errors  when  the  task  was  completed 
instead  of  when  the  error  occurred  (item  55,  Worksheet  3).  Faulty  performance 
was  identified  but  not  corrected  since  no  soldier  had  a  chance  to  try  the  task 
again  to  see  if  he/she  could  perform  without  the  error  (items  49  and  57,  Work¬ 
sheet  3) .  There  seems  to  have  been  a  resource  problem  with  this  administration 
of  the  lesson  in  that  only  four  assistant  instructors  were  available  (item  70, 
Worksheet  3) .  This  is  only  about  half  of  the  number  of  assistant  instructors 
needed  for  the  number  of  soldiers  being  trained.  A  revised  training  plan  that 
presents  clear  guidance  to  the  instructors  on  how  to  conduct  this  practice 
event,  on  what  to  look  for  during  soldier  practice,  on  how  to  identify  and  cor¬ 
rect  faulty  performance,  and  specifies  resource  requirements  and  tradeoffs  be¬ 
tween  time  allocation  and  number  of  instructional  personnel  may  eliminate  many 
of  these  practice  event  problems.  See  Practice  and  Feedback  under  DEMONSTRATION/ 
PRACTICE,  Section  VI,  RP  81-17.  Additional  guidance  is  contained  in  Section 
III,  PRACTICE,  and  Section  IV,  FEEDBACK,  In  RP  81-17. 
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TESTING  EVENTS.  The  test  for  this  block  was  administered  satisfactorily 
but  it  was  not  scored  in  a  consistent  manner.  Instructions  were  read  directly 
from  the  lesson  plan  (item  90,  RP  81-16),  soldiers  had  no  questions  on  what  the 
tasks  were  (item  91,  RP  81-16),  soldiers  had  no  questions  on  the  standards 
(item  93,  RP  81-16),  the  test  occurred  soon  after  training  to  minimize  forget¬ 
ting  (item  94,  RP  81-16),  all  tasks  were  tested  (item  95,  RP  81-16),  and 
examining  personnel  were  different  than  instructional  personnel  (item  103, 

RP  81-16).  When  it  came  to  scoring  soldier  performance,  examiners  did  not 
have  stopwatches  and  tested  two  soldiers  at  a  time  (item  98,  Worksheet  4). 

This  precluded  the  use  of  rigid  time  standards.  Seven  soldiers  were  given  a 
"GO"  when  they  had  exceeded  the  specified  time  standard  (item  104,  Worksheet  4). 
Thirty-six  soldiers  were  given  hints,  prompts,  or  cues  during  the  test  (item 
106,  Worksheet  4).  We  recommend  that  test  examiners  be  given  additional  training 
and  supervision  in  how  to  score  soldier  performance  in  a  consistent  manner. 

See  Contamination,  TESTING,  Section  VI,  RP  81-17. 
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SECTION  IX 


DO'S  AND  DON'TS  IN  CONDUCT  INC  A  TPE 

As  the  training  analyst  there  are  certain  things  that  you  should  and 
should  not  do  as  you  plan,  conduct,  and  analyze  the  results  of  the  training 
program  evaluation.  Some  of  these  have  already  been  mentioned  in  other  sec¬ 
tions  of  this  job  aid  but  are  reemphasized  here  because  of  their  importance  to 
the  success  of  the  overall  evaluation  effort. 

In  the  early  planning  stages  of  the  training  program  evaluation,  a  number 
of  organizations  may  be  involved.  Each  organization  will  be  vying  to  influence 
the  evaluation  in  a  direction  compatible  with  its  own  interests.  The  interests 
of  some  organizations  will  coincide  with  the  purposes  of  training  evaluation 
and  others  will  not.  You  must  take  an  active  role  in  the  early  planning  stages 
to  ensure  that  provisions  are  made  for  collecting  the  data  you  need  to  evaluate 
the  training  program.  Anything  that  hinders  you  in  collecting  the  necessary 
data  is  a  constraint;  you  should  try  to  keep  constraints  to  a  minimum.  One 
such  constraint  is  the  introduction  of  data  collection  requirements  that  are 
not  relevant  to  the  task  of  evaluating  training.  When  nonessential  data  col¬ 
lection  requirements  are  introduced,  the  planning  and  evaluation  processes  are 
complicated  and  the  energies  of  the  training  observers  are  diverted  to  the  col¬ 
lection  of  data  elements  that  are  of  little  use  in  evaluating  training.  For 
this  reason,  you  should  actively  oppose  the  efforts  of  any  organization  to  in¬ 
troduce  data  requirements  that  are  not  in  line  with  the  purpose  of  evaluating 
the  training  program. 

Section  III  of  this  job  aid  lists  some  items  you  will  need  in  conducting 
the  TPE  and  some  constraints  that  may  affect  the  conduct  of  the  TPE.  Among 
the  needs  listed  in  Section  III  is  the  need  for  well-trained,  highly  motivated 
training  observers  to  collect  the  TPE  data.  If  you  do  not  have  enough  observers, 
the  amount  of  information  that  you  will  be  able  to  provide  will  be  reduced  ac¬ 
cordingly.  You  should  make  it  very  clear  to  the  sponsor  of  the  evaluation  what 
you  can  and  cannot  do,  depending  on  the  availability  of  training  observers. 
Emphasize  that  you  will  need  the  observers  well  in  advance  to  allow  plenty  of 
time  for  training  and  selecting  your  data  collectors  before  training  begins. 

Arrange  to  have  the  observers  working  for  you  directly.  Among  the  advan¬ 
tages  of  having  the  data  collectors  report  directly  to  you  are:  (1)  you  can 
schedule  which  observers  collect  which  data  and  therefore  be  constantly  aware 
of  who  is  collecting  what  data  and  when  and  where  it  is  being  collected;  (2) 
your  daily  contact  with  the  observers  will  provide  additional  information 
about  who  is  using  the  TPE  worksheets  well  and  who  needs  additional  training; 

(3)  since  the  observers  are  reporting  to  you  directly,  you  are  more  likely  to 
receive  the  completed  TPE  worksheets  promptly  and  you  may  receive  additional 
information  in  talking  to  the  observers  that  was  not  recorded  on  the  worksheets; 
and  (A)  you  are  in  position  to  influence  the  conditions  under  which  the  ob¬ 
servers  work  and  thus  act  to  minimize  the  factors  that  may  have  an  adverse 
effect  on  observer  motivation. 
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The  success  or  failure  of  the  training  program  evaluation  that  you  conduct 
will  depend  to  a  large  extent  on  the  ability,  skill  and  motivation  of  the  in¬ 
dividuals  who  will  be  observing  the  training  and  testing  and  collecting  the 
TPE  data.  You  are  strongly  urged  to  conduct  a  TPE  workshop  for  the  observers 
(see  Section  V  of  this  research  product)  prior  to  the  beginning  of  the  training 
program.  As  an  absolute  minimum,  all  observers  should  have  had  supervised 
practice  in  using  the  worksheets  prior  to  any  involvement  in  an  actual  evalua¬ 
tion. 

In  order  to  select  the  individuals  who  will  make  the  best  training  ob¬ 
servers,  you  should  watch  the  observers  as  they  collect  data  during  the  prac¬ 
tice  portion  of  the  TPE  workshop,  and  review  their  completed  worksheets  to 
determine  how  well  they  recorded  the  required  observations.  You  will  find  that 
some  individuals  either  do  not  attend  to  the  class  being  conducted  or  attend 
but  record  very  little  useful  information  on  the  TPE  worksheets.  Such  persons 
make  poor  observers  and  should  not  be  used  during  the  actual  program  evaluation 
unless  no  one  else  is  available.  Other  persons  will  attend  to  the  class  very 
intently  and  become  so  wrapped  up  in  what  is  being  taught  (i.e.,  the  subject 
matter  of  the  class)  that  they  pay  little  attention  to  how  training  is  con¬ 
ducted.  If  they  record  any  comments,  the  comments  will  be  about  the  technical 
content  of  the  class  and  the  training  and  testing  processes  will  be  ignored 
for  the  most  part.  Subject  matter  experts  often  respond  in  this  manner  when 
asked  to  observe  training.  For  this  reason,  subject  matter  experts  typically 
do  not  make  very  good  TPE  observers.  Finally,  there  are  those  select  few  who 
watch  training  closely  and  record  many  useful  objective  comments  in  response 
to  the  items  listed  on  the  TPE  worksheets.  These  are  the  persons  that  you 
should  select  to  collect  the  bulk  of  the  TPE  data. 

In  the  workshop,  you  should  try  to  motivate  the  observers  by  explaining 
the  importance  of  training  evaluation  and  by  stressing  the  role  of  the  observer 
in  collecting  the  necessary  data  for  the  evaluation.  You  should  also  prepare 
the  observers  for  the  cool  reception  that  they  might  receive  from  instructional 
personnel  by  explaining  that  training  evaluators  are  often  perceived  as  adver¬ 
saries  by  the  training  deliverer.  In  planning  the  evaluation,  you  should  keep 
the  observers'  interests  in  mind  when  making  decisions  that  will  impact  on  the 
observers.  For  example,  you  should  avoid  over-committing  your  training  ob¬ 
servers  by  limiting  the  number  of  data  elements  that  they  are  required  to  col¬ 
lect  at  any  one  time  and  the  number  of  hours  of  instruction  that  they  will 
observe  on  any  given  day.  In  addition  the  data  collection  plan  should  make 
provisions  for  the  well-being  and  comfort  of  the  data  collectors.  Such  pro¬ 
visions  might  include  transportation  to  and  from  the  training  site,  messirg 
facilities  near  the  training  site  and  the  opportunity  to  use  them,  and  a  tem¬ 
porary  shelter  where  they  may  go  from  time  to  time  to  escape  the  elements  when 
training  is  conducted  outside  in  inclement  weather  (since  they  may  very  well  be 
out  in  the  inclement  weather  much  longer  than  any  of  the  instructors  or  soldiers). 

In  planning  and  conducting  an  evaluation,  you  will  be  coordinating  your 
activities  with  representatives  of  several  different  groups.  These  include 
other  evaluation  agencies,  the  organization  that  is  sponsoring  your  involvement 
in  the  evaluation,  and  the  group  responsible  for  developing  and  perhaps  de¬ 
livering  the  training  program.  Other  than  the  sponsoring  organization,  your 
most  critical  association  is  with  the  group  that  develops  and  delivers  the  in¬ 
struction.  It  is  important  that  you  establish  as  much  rapport  as  possible  with 
this  group  early  in  the  planning  process.  This  may  be  difficult  since  training 
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developers  and  deliverers  often  perceive  training  evaluation  as  an  annoyance 
and  perhaps  even  a  threat.  There  are  things  you  can  say  and  do  that  can  reduce 
the  extent  to  which  the  evaluation  is  perceived  as  a  threat.  For  example,  you 
can  show  the  training  developers  and  deliverers  the  worksheets  you  will  be 
using  to  evaluate  training  and  tell  them  they  may  inspect  the  completed  work¬ 
sheets  if  they  wish.  Explain  to  the  instructors  that  they  are  not  personally 
being  evaluated,  but  that  it  is  the  training  program  that  is  being  evaluated. 
Explain  to  the  training  developers  that  you  would  like  to  work  with  them  to 
obtain  the  best  program  possible,  that  they  will  be  the  first  to  see  any  criti¬ 
cism  of  the  program  or  recommendations  for  program  changes,  and  that  the  data 
that  you  collect  will  not  be  given  to  persons  or  organizations  that  would  use 
it  to  "hang  them."  All  of  this  requires  that  you  interact  directly  with  the 
training  developer  and  deliverer.  It  is  especially  important  that  you  establish 
an  open  line  of  communications  with  a  person  on  the  training  development  team 
at  the  level  where  the  training  is  designed  and  changes  in  the  program  are 
instituted;  you  should  establish  this  vital  communications  link  as  early  in 
the  program  as  possible,  preferably  during  the  early  stages  of  program  develop¬ 
ment.  In  this  way  your  recommendations  for  changes  in  the  training  program 
will  be  sure  to  reach  the  persons  who  are  responsible  for  making  the  changes  in 
a  timely  manner.  When  changes  are  made  in  the  training  program,  you  can  obtain 
information  about  the  changes  from  this  same  individual.  Do  not  rely  on  the 
sponsor  of  the  evaluation  to  act  as  a  go-between  between  you  and  the  program 
change  agent. 

Arrangement  for  distribution  of  the  various  data  elements  should  be  made 
during  the  planning  for  the  program  evaluation.  Make  it  clear  before  evalua¬ 
tion  begins  what  data  will  be  provided  to  the  sponsor,  and  when  in  relation  to 
the  program  they  will  be  provided. 

You  may  wish  to  share  data  with  other  evaluation  agencies,  but  do  not 
enter  into  any  data  sharing  agreements  without  first  consulting  the  organiza¬ 
tion  responsible  for  developing  and  delivering  the  training  program  and  the 
sponsor  of  the  evaluation.  If  you  are  sharing  the  raw  data  with  other  agencies, 
arrange  to  get  what  you  need  from  the  data  first,  make  copies  for  the  request¬ 
ing  organization,  and  keep  the  original  data  on  file  for  ease  of  reference. 

A  carefully  planned  evaluation  following  the  guidance  in  the  paragraphs 
above  and  in  Section  III  of  this  report  will  help  to  ensure  the  success  of  youi 
training  program  evaluation  effort.  But  it  is  only  half  of  the  ball  game. 

The  training  program  evaluation  must  still  be  carried  out  during  the  program 
implementation.  One  of  your  tasks  as  the  training  analyst  is  to  keep  the  evalu¬ 
ation  on  track,  seeing  to  it  that  the  direction  and  intent  of  the  evaluation 
is  not  compromised.  As  the  training  program  gets  underway,  some  of  the  evalu¬ 
ation  agencies  may  decide  that  they  need  certain  additional  data  and  seek  to 
introduce  additional  data  collection  instruments  for  the  observers  to  complete. 
Resist  attempts  to  have  your  observers  complete  additional  data  collection 
forms.  New  data  collection  instruments  should  be  introduced  only  when  it  can 
be  shown  that  the  data  is  needed  to  provide  answers  to  the  sponsor's  questions 
that  would  otherwise  go  unanswered.  If  additional  data  collection  instruments 
are  introduced,  you  may  have  to  forego  collecting  some  other  data  that  you  had 
planned  to  collect  so  as  not  to  overload  the  observers. 


You  should  provide  the  data  and  other  information  that  you  agreed  to  pro¬ 
vide  to  the  sponsoring  agency  according  to  the  schedule  that  was  mutually 
agreed  upon  during  planning.  To  enable  you  to  do  this,  the  sponsoring  agency 
must  provide  you  with  the  material  and  personnel  resources  that  they  agreed  to 
provide.  If  the  sponsoring  agency  decides  in  the  middle  of  the  evaluation  that 
more  data  are  needed,  then  you  must  ensure  that  the  resources  necessary  to  col¬ 
lect  that  data  will  be  made  available. 

Occasionally  you  may  find  that  the  instructors  delivering  the  training  or 
their  supervisors  start  to  look  upon  the  training  observers  as  training  distrac- 
tors  after  training  has  been  going  on  for  a  short  time  and  try  to  restrict  the 
observers'  access  to  some  training  or  testing  events.  This  should  not  occur  if 
you  have  properly  trained  the  observers  and  have  been  up-front  with  the  train¬ 
ing  deliverers  letting  them  know  exactly  what  the  observers  will  be  doing  and 
the  use  to  which  the  data  will  be  put.  If  the  instructors  do  attempt  to 
restrict  access  to  the  training  or  testing  events,  you  must  take  immediate 
action  or  important  data  will  be  lost.  You  should  first  try  to  find  out  why 
the  observers  have  been  restricted  and  work  something  out  with  the  training 
deliverer.  If  no  agreement  can  be  reached  then  you  may  have  to  involve  the 
sponsor  of  the  evaluation  to  bring  pressure  upon  the  training  deliverer. 

If  the  training  observers  are  working  for  you  directly  as  recommended,  the 
completed  worksheets  will  be  returned  to  you  personally  by  the  observers  shortly 
after  the  completion  of  each  block  of  instruction.  After  receiving  the  work¬ 
sheets  for  two  or  three  blocks  of  instruction,  you  may  notice  that  while  some 
observers  provide  detailed  information  about  the  training  that  they  observed, 
others  observing  the  same  training  may  have  little  more  than  a  check  mark  be¬ 
side  each  worksheet  item.  When  you  find  that  observers  are  providing  little 
useful  information,  you  should  investigate  to  determine  the  nature  of  the 
problem.  You  may  find  in  talking  with  the  observer  that  he  or  she  can  tell 
you  about  the  class  in  detail,  but  simply  failed  to  record  the  details  on  the 
worksheet.  In  this  case  you  must  encourage  the  observer  to  record  the  neces¬ 
sary  data.  If  the  observer  cannot  provide  much  information  about  the  class, 
you  may  have  to  monitor  that  observer's  activities  to  isolate  the  problem. 

The  observer  may  require  additional  training  in  TPE  or  observation  techniques, 
or  lack  the  motivation  to  carefully  observe  training  and  record  the  required 
data.  If  the  problems  with  the  observer  cannot  be  solved,  the  observer  may 
have  to  be  replaced. 

Even  when  the  data  provided  by  the  observers  is  entirely  satisfactory,  it 
is  useful  for  you  to  monitor  some  blocks  of  instruction  or  portions  of  those 
blocks  in  order  to  get  a  first-hand  view  of  the  conduct  of  the  training  program. 
Your  observations  may  identify  some  training  problems  that  the  observers  missed 
or  failed  to  record.  Your  presence  during  the  training  also  indicates  to  the 
observers  and  instructors  that  you  have  more  than  a  passing  interest  in  the 
training  program  and  the  evaluation  activities  of  the  observers.  In  your 
visits  to  the  training  site,  ask  the  observers  if  they  are  encountering  any 
problems  and  talk  to  the  instructors  or  their  supervisors,  if  possible,  about 
bow  Mi  in  (nr.  i  going.  The  informal  ion  von  gain  by  visiting  the  training 
1 1  c;,  ami  iiuuiicoi  mg  training  will  ottrii  prove  invaluable. 
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As  the  data  are  received  on  each  block  of  instruction  you  will  summarize 
and  analyze  the  data,  providing  recommendations  for  changes  in  the  training 
program  to  the  training  developer  and  Information  about  the  effectiveness  and 
efficiency  of  the  program  to  the  sponsor  of  the  evaluation.  Suggestions  for 
changes  in  the  training  program  should  be  submitted  (within  24  hours  of  the 
time  that  the  block  of  Instruction  was  conducted)  directly  to  the  person  on 
the  training  development  team  responsible  for  making  program  changes.  Your 
suggestions  to  the  training  developer  should  be  in  the  form  of  a  short  memoran¬ 
dum  that  clearly  identifies  the  block  of  instruction  observed,  the  training 
problems  observed,  and  your  recommendations  for  correcting  the  problems.  The 
training  developer  to  whom  the  suggestions  are  submitted  should  then  check  into 
the  possibility  of  making  the  recommended  changes  and  inform  you  which  changes 
were  made  and  which  were  not.  The  Information  provided  to  the  sponsor  of  the 
evaluation  regarding  the  effectiveness  and  efficiency  of  the  training  program 
will  be  based  on  the  same  data,  but  the  sponsor  may  not  need  or  want  detailed 
information  on  each  block  of  instruction.  Feedback  to  the  sponsor  may  take 
the  form  of  short  reports  identifying  recurring  problems,  submitted  at  regular 
intervals  as  the  program  is  being  conducted  and/or  a  final  report  submitted  on 
completion  of  the  training  program. 

The  conclusions  that  you  draw  and  the  recommendations  that  you  make  during 
an  evaluation  can  be  no  better  than  the  data  upon  which  they  are  based. 

Although  the  TPE  system  is  designed  to  provide  accurate  objective  data  when 
used  correctly,  it  can  lead  to  erroneous  conclusions  if  misused.  If  you  as 
the  analyst  omit  many  of  the  more  important  items  from  the  worksheets  or  allow 
your  observers  to  use  the  TPE  worksheets  merely  as  checklists  without  making 
any  comments,  the  data  is  likely  to  paint  a  rosy  but  inaccurate  picture  of  the 
efiectiveness  of  the  training  program.  If  the  observers  hesitate  to  record 
what  they  observe  because  they  are  inhibited  by  the  instructors  or  fear  reper¬ 
cussions  from  higher  authorities,  the  usefulness  of  the  data  is  greatly  dimin¬ 
ished.  Similarly  if  you  temper  the  conclusions  that  you  draw  about  the  train¬ 
ing  program  because  you  do  not  want  to  ruffle  the  feathers  of  some  individual 
or  agency,  your  effectiveness  as  a  training  analyst  is  greatly  reduced.  The 
objectivity  of  both  the  observer  and  the  analyst  must  be  maintained  in  order 
for  TPE  to  work.  For  this,  you  will  need  unqualified  support  at  command  level. 
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APPENDIX  A 


WORKSHEET  1,  TRAINING  PLAN  -  Part  I,  Equipment  and  Materials 
Class/Lesson  Title _ 


Training  Observer _ _ '  _ 

1.  List  under  the  appropriate  heading  below  the  training  aids,  equipment,  and 
materials  that  will  be  used  during  this  lesson. 

Training  Equipment  Training  Materials  Training  Aids 


2.  List  any  job  aids  that  will  be  provided  to  soldiers  and  used  during  this 
training. 


3.  From  the  lesson  plan,  briefly  describe  the  characteristics  required  of  the 
training  site.  Include  how  the  site  should  be  prepared  for  training. 


4.  List  safety  precautions  that  should  be  emphasized  and  followed  during  this 
training. 


WORKSHEET  1,  TRAINING  PLAN  -  Part  II,  Training  Objectives 
Lesson  Title  ___ _ _ 


TRAINING  OBJECTIVES 


WORKSHEET  1,  TRAINING  PLAN  -  Part  III,  Training  Events 
Lesson  Title _ 


T 


TRAINING  EVENTS 

- - - r 


Event 

Type 


Events 


Occur 
Yes  I  No 


Comments 


APPENDIX  B 


DEVELOPING  QUESTIONNAIRES  FOR  SOLDIERS  AND  INSTRUCTORS 

Occasionally  you  will  be  called  upon  to  evaluate  training  but  will  have 
neither  the  time  nor  the  resources  to  follow  the  procedures  recommended  in  this 
job  aid.  In  lieu  of  conducting  a  rigorous  TPE  or  as  a  supplement  to  the  pro¬ 
cedures  outlined  in  this  job  aid,  you  may  wish  to  evaluate  training  by  ques¬ 
tioning  soldiers  and/or  instructors  concerning  their  training  experiences  or 
their  perceptions  about  the  training.  If  you  decide  to  use  this  approach,  you 
will  need  to  construct  questionnaires,  so  that  the  same  questions  are  asked  of 
each  soldier  or  instructor.  Although  it  may  seem  a  simple  matter  to  write  a 
few  questions  for  soldiers  to  answer  about  the  quality  of  the  training  that 
they  received,  developing  a  questionnaire  that  provides  valid,  meaningful  data 
is  a  rather  difficult  task.  If  you  find  it  necessary  to  develop  a  question¬ 
naire  for  training  evaluation  purposes,  it  is  suggested  that  you  seek  technical 
assistance  from  someone  who  has  expertise  in  designing  questionnaires.  If 
technical  assistance  is  not  available,  the  guidance  in  this  Appendix  should 
keep  you  from  making  any  serious  mistakes  in  developing  questionnaires. 

Do  not  ask  soldiers  or  instructors  for  their  general  impressions,  or 
opinions  about  training.  Do  not  ask  soldiers  to  rate  the  quality  of  the  train¬ 
ing  that  they  received.  These  kinds  of  questions  do  not  provide  the  kinds  of 
data  that  you  will  need  as  an  evaluator.  Ask  only  questions  that  the  respon¬ 
dent  (the  person  filling  out  the  questionnaire)  has  a  basis  for  answering  —  a 
basis  resting  in  his  or  her  recent  experiences  during  the  training  program. 

If  both  soldiers  and  instructors  are  questioned,  the  questions  should  be  dif¬ 
ferent  for  the  two  groups. 

The  guidance  included  below  will  help  you  construct  questionnaires  when 
you  must  use  the  questionnaire  method.  Following  the  guidance  in  this  Appendix 
will  help  you  construct  better  questionnaires,  but  even  good  questionnaires 
are  unlikely  to  provide  data  that  are  comparable  to  that  provided  by  other 
methods  described  in  this  report.  Therefore,  questionnaire  methods  should  only 
be  used  as  a  last  resort  and  the  resulting  data  Interpreted  with  great  care. 

Questioning  Soldiers  About  the  Formal  Instruction  They  Received 

If  you  cannot  put  an  observer  at  a  training  event  but  need  data  on  the 
conduct  of  that  event,  you  can  question  the  soldiers  who  were  present  for  the 
training.  You  can  use  their  memories  to  get  information  on  what  happened. 
Remember  that  their  memories  are  going  to  be  faulty.  You  will  have  to  sample 
several  (but  not  all)  soldiers  to  get  a  stable  answer  to  any  question.  It  is 
not  necessary  to  ask  each  soldier  sampled  the  same  set  of  questions. 

You  will  want  to  construct  most  of  these  questions  so  that  they  can  be 
answered  "yes"  or  "no."  Do  not  expect  the  soldiers  to  give  you  more  detailed 
answers,  from  memory,  than  you  expect  from  your  observer,  on-the-spot.  Ask 
questions  in  words  and  about  experiences  that  have  meaning  for  them. 

Do  not  ask  too  many  questions  of  any  soldier.  Ten  to  15  questions  is 
plenty.  If  you  have  more  than  that,  get  two  samples  of  soldiers.  Give  some 
questions  to  one  sample  and  the  remaining  questions  to  the  other  sample. 


Some  examples  of  the  kinds  of  questions  soldiers  can  be  asked  are  below. 
Note  that  these  questions  parallel  the  Items  in  RP  81-16. 

Were  instructors  available  when  you  needed  them? 

Did  you  have  a  _ _  (tool,  handout,  piece  of 

equipment)  when  you  needed  it? 

Did  you  have  your  own  copy  of  _  (handout)? 

Did  you  have  any  trouble  _____  (bearing,  seeing)? 

Do  you  know  what  _ _ _  (a  term  or  concept)  means? 

Was  the  _  (task,  subtask,  step)  demonstrated  for 

you  by  the  instructor? 

Could  you  _ _  (see»  hear,  understand)  the  demon¬ 
stration  of  _ (task,  subtask,  step)? 

Did  you  get  a  chance  to  practice  the  _  (task,  subtask, 

step)? 

Did  an  assistant  instructor  watch  you  during  practice,  correcting 
your  errors  as  you  went  along? 

Questioning  Soldiers  About  Their  Proficiency  and  Events  Occurring  Outside 
Formal  Training 

If  you  want  to  know  how  confident  soldiers  are  that  they  can  perform  the 
training  tasks,  or  what  they  know  about  events  occurring  outside  of  the  formal 
training  situation,  you  can  ask  a  different  form  of  question.  Be  careful  here, 
however.  Confidence  ratings  have  not  been  shown  to  relate  well  to  proficiency. 
Soldiers  tend  to  give  unrealistically  high  estimates  of  their  ability  to  perform 
the  tasks  in  question. 

Soldiers  can  answer  questions  about  what  is  happening  to  them  because 
they  are  in  a  training  environment.  You  can  question  them  about  such  things 
as  rumors,  things  that  happen  during  non-training  time  (phantom  training, 
rewards/punishments  for  good/poor  performance,  etc.),  what  they  know  about  ad¬ 
ministrative  matters  (post-training  assignments,  bonuses,  UCMJ  Articles,  pay 
and  allowances,  etc.),  and  any  other  item  on  which  they  have  Information  or 
misinformation. 

Such  items  must  be  very  carefully  worded  so  that  they  mean  the  same  thing 
to  each  soldier.  Keep  the  questions  simple. 


Avoid  two  questions  in  one  sentence.  Keep  the  number  of  answer  categories 
small  (throe  or  less). 


As  a  result  of  this  training,  Z  can  perform  the  duties  of  a  loader  or 
driver  on  an  M60A1  Tank. 


Strongly  Agree 
Agree 
Uncertain 
Disagree 

Strongly  Disagree 


BAD.  Two  questions 
in  one;  too  many 
answer  categories. 

(Note:  Agree,  Uncertain, 
Disagree  is  better.) 


I  know  how  to  be  a  loader  on  an  A1  Tank. 


Yes  BETTER.  One 

Need  More  Training  First  question;  three 

No  answer  categories; 

simpler  wording. 


The  NBC  training  1  received  was 


I  understood 


Outstanding 

Excellent 

Fair 

Satisfactory 

Unsatisfactory 


Most 

Some 

None 


of 


the  NBC  spot 


BAD.  The  soldier  has 
no  basis  for  making 
a  training  quality 
judgment;  NBC  training 
may  have  had  high  and 
low  spots;  the  usual 
soldier  cannot 
differentiate  between 
adjacent  answer 
categories;  too  many 
answer  categories. 

report  training. 


BETTER.  Sharper  focus 
for  the  soldier; 
three  answer  categories. 


Ue  have  been  getting  extra  training  from  the  unit  cadre  after  regular 
training. 


Yes  _  BAD.  This  doesn't  give 

No  _____  enough  information. 

Did  you  get  to  practice  assembly/disassembly  of  the  M240  Machlnegun  after 
regular  training? 


Yes  _____  (How  many  times?  _ ) 

No  _ 

BETTER.  Sharper  focus; 
some  information  on 
amount  of  practice. 
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Do  not  ask  test  questions  (What  is  the  effective  range  of  the  M240  Machine- 
gun?),  questions  covered  under  the  provisions  of  the  Privacy  Act  (Has  there 
been  any  homosexual  activity  in  your  barracks?),  or  questions  with  a  heavy 
social  or  moral  tone  (How  many  members  of  your  platoon  smoke  pot?). 

uestioning  Instructors  About  the  Instruction  They  Conducted 


Instructors  can  add  to  your  knowledge  about  "what"  happened  during  train¬ 
ing  from  their  perspective.  Most  of  what  was  said  above  concerning  questioning 
soldiers  about  "what"  happened  is  applicable  here. '  Instructors,  however,  can 
give  you  more  information  and  you  can  ask  more  questions.  You  can  also  get 
written  answers  (you  do  not  have  to  stick  to  simple  "yes/no"  answers).  Examples 
of  the  kinds  of  questions  you  can  ask  instructors  follow  below. 

Did  you  have  enough  assistant  instructors  to  cover  the  stations 
during  practice? 

Did  you  have  what  you  needed  to  conduct  this  training?  (If  "no," 
what  was  missing?) 

What  problems  were  there  with  the  site  selected  for  this  training 
(light,  noise,  facilities,  etc.)? 

Did  the  weather  affect  training?  (If  "yes,"  how?) 

Do  you  need  more  (less)  time  for  this  block  of  instruction?  (If 
"yes,"  why?) 

Did  most  of  the  soldiers  (90%  or  more)  reach  standard  on  all  the 
tasks  during  training?  (If  "no,"  why  not?) 

Did  you  follow  the  lesson  plan?  (If  "no,"  why  not?) 

Are  the  training  aids  for  this  block  accurate  and  up-to-date?  (If 
"no,"  what  changes  are  needed?) 
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APPENDIX  C 


STATION  1 


Soldier  Engagement  Time  Hit 


Number/Type* 

(Sec) 

Miss 

1/BM 

2/BM 

52 

Miss 

3/BM 

47 

Miss 

4/BM 

56 

Hit 

5/BSA 

74 

Miss 

6/ BSA 

65 

Miss 

7/BSA 

59 

Hit 

1/BM 

34 

Miss 

1/BM 

39 

Miss 

3/BM 

42 

Hit 

4/BSA 

41 

Miss 

5/BSA 

46 

Miss 

6/BSA 

39 

Hit 

1/BM 

2/BM 

78 

Miss 

3/BM 

81 

Miss 

4/BM 

66 

Miss 

5/BM 

50 

Hit 

6/BM 

43 

Hit 

Instructor  Comments 

Aborted  by  instructor.  Soldier  did  not 
know  how  to  use  ballistic  reticle. 

Wrong  range,  wrong  lead. 

Wrong  range. 

Wrong  sensing,  wrong  adjustment. 

Wrong  adjustment. 

Wrong  range,  wrong  lead. 

Wrong  range. 

Wrong  adjustment. 

Wrong  adjustment. 

Aborted  by  instructor.  Soldier  did  not 
know  how  to  use  ballistic  reticle. 

Wrong  range,  wrong  lead. 

Wrong  range,  wrong  lead. 

Wrong  range,  wrong  lead. 


*  BS  -  Ballistic  Reticle,  Stationary 
NBS  -Non-Ball is tic  Reticle,  Stationary 
BM  -  Ballistic  Reticle,  Moving  Target 
NBM  -  Non-Ballistic  Reticle,  Moving  Target 
BSA  -  Ballistic  Reticle,  Sense  and  Adjustment 
SBSA  -  Non-Ballistic  Reticle,  Sense  and  Adjustment 


STATION  1  (Cont) 


Soldier 

Engagement 

Number/Type 

Time 

(Sec) 

Hit 

Miss 

Instructor  Comments 

4 

1/BM 

29 

Miss 

Wrong  lead. 

2/BM 

22 

Hit 

3/BSA 

46 

Miss 

Wrong  adjustment. 

4/BSA 

38 

Miss 

Wrong  adjustment. 

5/BSA 

38 

Miss 

Wrong  adjustment. 

5/BSA 

25 

Hit 

5 

1/BM 

26 

Miss 

Wrong  lead. 

2/BM 

34 

Miss 

Wrong  lead. 

3/BM 

32 

Hit 

4/BSA 

39 

Hit 

6 

1/BM 

37 

Miss 

Wrong  range,  wrong  lead. 

2/BM 

39 

Miss 

Wrong  range,  wrong  lead. 

3/BM 

46 

Hit 

4/BSA 

62 

Miss 

Wrong  adjustment. 

5/BSA 

66 

Miss 

Wrong  sensing,  wrong  adjustment 

6/BSA 

64 

Hit 

STATION 

2 

1 

1/NBS 

19 

Miss 

Wrong  sight  picture. 

2/NBS 

22 

Hit 

3/NBM 

34 

Miss 

Wrong  lead. 

4/NBM 

39 

Miss 

Wrong  lead. 

5/NBSA 

50 

Miss 

Wrong  adjustment. 

0/NI5SA 

rXi 

Miss 

Wrong  adjustment. 

STATION  2  (Cont) 


Soldier  engagement 
Number/Type 

Time 

(Sec) 

11U 

Miss 

Just  rue tor  Comments 

1  7/BS 

34 

Mis  8 

Wrong  range. 

8/BS 

35 

Miss 

Wrong  range. 

9/BM 

77 

Miss 

Wrong  range,  wrong  lead. 

10/BM 

84 

Miss 

Wrong  range,  wrong  lead. 

11/BSA 

66 

Miss 

Wrong  adjustment. 

12/BSA 

83 

Miss 

Wrong  adjustment. 

2  1/NBS 

26 

Hit 

2/NBS 

21 

Hit 

3/NBM 

39 

Miss 

Wrong  lead. 

4/NBM 

30 

Hit 

5/NBSA 

61 

Hit 

6/NBSA 

30 

Hit 

7/BS 

54 

Miss 

Wrong  range. 

8/BS 

51 

Miss 

Wrong  range. 

9/BM 

48 

Miss 

Wrong  range,  wrong  lead. 

10/BM 

35 

Miss 

Wrong  lead. 

11/BSA 

47 

Miss 

Wrong  adjustment. 

12/BSA 

58 

Hit 

3  1/NBS 

55 

Hit 

2/NBS 

40 

Hit 

3/NBM 

46 

Miss 

Wrong  lead. 

4/NBM 

45 

Hit 

5/NBSA 

62 

Hit 

STATION  2  (Cont) 


S 


Soldier 

Engagement 

Number/Type 

Time 

(Sec) 

Hit 

Miss 

Instructor  Comments 

3 

6/NBSA 

50 

Hit 

5 

7/BS 

37 

Miss 

Wrong  range. 

8/BS 

21 

Hit 

►  ' 

9/BM 

35 

Hit 

STATION 

3 

1 

1/BS 

26 

Miss 

Wrong  range. 

2/BS 

55 

Hit 

1 

•  . 

3/BM 

74 

Miss 

Wrong  lead. 

• 

4/BM 

67 

Hit 

5/BM 

60 

Hit 

s 

* 

6/BSA 

84 

Hit 

2 

1/BS 

14 

Hit 

2/BM 

32 

Hit 

5 

3/BM 

27 

Hit 

4/BSA 

36 

Miss 

Wrong  adjustment. 

5/BSA 

42 

Hit 

:  i 

3 

1/BS 

25 

Miss 

Wrong  range. 

2/BS 

37 

Miss 

Wrong  range. 

3/BS 

35 

Miss 

Wrong  range. 

» 

*• 

4/BS 

34 

Miss 

Wrong  range. 

5/BS 

38 

Hit 

6/BM 

94 

Miss 

Wrong  range,  wrong  lead. 

r 

7/BM 

79 

Miss 

Wrong  range,  wrong  lead. 

i 
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V 


STATION  3  (Cont) 


Soldier  Engagement 
Number/Type 

Time 

(Sec) 

Hit 

Miss 

3  8/BM 

90 

Miss 

9/BM 

82 

Miss 

10/BM 

70 

Miss 

11/BM 

88 

Miss 

12/BSA 

94 

Miss 

13/BSA 

85 

Miss 

14/BSA 

89 

Miss 

15/BSA 

64 

Miss 

16/BSA 

69 

Miss 

17/BSA 

76 

Hit 

4  1/BS 

28 

Miss 

2/BS 

35 

Hit 

3/BM 

38 

Miss 

4/BM 

30 

Miss 

5/BM 

42 

Hit 

6/BSA 

40 

Miss 

7/BSA 

65 

Hit 

8/BSA 

53 

Hit 

STATION  5 

1  1/NBM 

27 

Miss 

2/NBM 

29 

Miss 

3/NBM 

24 

Hit 

4/BM 

Instructor  Comments 

Wrong  range,  wrong  lead. 

Wrong  range. 

Wrong  range. 

Wrong  range,  wrong  lead. 

Wrong  sensing,  wrong  adjustment. 
Wrong  sensing,  wrong  adjustment. 
Wrong  adjustment. 

Wrong  adjustment, 
itfrosog  adjustment. 

Wrong  range. 

Wrong  range,  wrong  lead. 

Wrong  lead. 

Wrong  adjustment. 


Wrong  lead. 

Wrong  lead. 

Aborted  by  instructor.  Soldier  did  not 
know  how  to  use  ballistic  reticle. 
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STATION  5  (Cont) 


Soldier 

Engagement 

Number/Type 

Time 

(Sec) 

Hit 

Miss 

Instructor  Comments 

1 

5/BM 

75 

Miss 

Wrong 

range. 

wrong 

lead. 

6/BM 

84 

Miss 

Wrong 

range, 

wrong 

lead. 

7/BM 

74 

Miss 

Wrong 

lead. 

8/BS 

66 

Hit 

9/BS 

51 

Hit 

10/ BM 

64 

Miss 

Wrong 

lead. 

11/BM 

68 

Miss 

Wrong 

lead. 

12 /BM 

74 

Hit 

13/BM 

60 

Hit 

2 

1/NBM 

25 

Hit 

2/BM 

49 

Hit 

3/BSA 

46 

Hit 

4/BSA 

28 

Hit 

3 

1/NBM 

37 

Miss 

Wrong 

lead. 

2/NBM 

46 

Hit 

3/BM 

51 

Miss 

Wrong 

range. 

wrong 

lead. 

4/BM 

40 

Miss 

Wrong 

range. 

wrong 

lead. 

5/BM 

27 

Hit 

6/BSA 

32 

Hit 

7/BSA 

48 

Hit 

4 

1/NBM 

18 

Miss 

Wrong 

lead. 

2/NBM 

28 

Hit 

3/BM 

36 

Miss 

Wrong 

range. 

wrong 

lead . 

STATION  5  (Cont) 


Soldier 

Engagement 

Number/Type 

Time 

(Sec) 

Uil 

Miss 

Inst  nicUH  IaMiuiumU  s 

4 

4/BM 

40 

Miss 

Wrong  range,  wrong  lead. 

5/BM 

52 

Hit 

6/BSA 

38 

Miss 

Wrong  sensing,  wrong  adjustment. 

7/BSA 

29 

Miss 

Wrong  adjustment. 

8/BSA 

47 

Hit 

5 

1/NBM 

25 

Hit 

2/BM 

34 

Miss 

Wrong  range,  wrong  lead. 

3/BM 

43 

Hit 

4/BSA 

64 

Hit 

5/BSA 

67 

Hit 

